You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/12/01 23:12:05 UTC

[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 690 - Failure

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/690/

2 tests failed.
REGRESSION:  org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestDemoParallelLeafReader

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)




Build Log:
[...truncated 1679 lines...]
   [junit4] Suite: org.apache.lucene.index.TestDemoParallelLeafReader
   [junit4]   2> ??? 01, 2014 10:21:13 ? com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestDemoParallelLeafReader
   [junit4]   2> ==== jstack at approximately timeout time ====
   [junit4]   2> "TEST-TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField-seed#[1A30B36FF7E830A0]" ID=22 RUNNABLE
   [junit4]   2> 	at sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
   [junit4]   2> 	at sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:71)
   [junit4]   2> 	at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
   [junit4]   2> 	at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:136)
   [junit4]   2> 	at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:149)
   [junit4]   2> 	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:175)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
   [junit4]   2> 	at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:46)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
   [junit4]   2> 	at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
   [junit4]   2> 	at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
   [junit4]   2> 	at java.nio.channels.FileChannel.open(FileChannel.java:287)
   [junit4]   2> 	at java.nio.channels.FileChannel.open(FileChannel.java:334)
   [junit4]   2> 	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:205)
   [junit4]   2> 	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
   [junit4]   2> 	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:654)
   [junit4]   2> 	- locked org.apache.lucene.store.MockDirectoryWrapper@3f04f430
   [junit4]   2> 	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
   [junit4]   2> 	at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:91)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4504)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:509)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:476)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:514)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:379)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1398)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   2> 	at org.apache.lucene.index.TestDemoParallelLeafReader$3.reindex(TestDemoParallelLeafReader.java:823)
   [junit4]   2> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader.getParallelLeafReader(TestDemoParallelLeafReader.java:394)
   [junit4]   2> 	- locked org.apache.lucene.index.TestDemoParallelLeafReader$3@1b66d903
   [junit4]   2> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$1.warm(TestDemoParallelLeafReader.java:133)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4097)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   2> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2> 	- locked org.apache.lucene.index.SerialMergeScheduler@3519b184
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   2> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   2> 	at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> "SUITE-TestDemoParallelLeafReader-seed#[1A30B36FF7E830A0]" ID=21 RUNNABLE
   [junit4]   2> 	at sun.management.ThreadImpl.dumpThreads0(Native Method)
   [junit4]   2> 	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:671)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$900(ThreadLeakControl.java:62)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:412)
   [junit4]   2> 	- locked java.lang.Object@29d56c58
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:642)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:129)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:559)
   [junit4]   2> 
   [junit4]   2> "JUnit4-serializer-daemon" ID=9 TIMED_WAITING
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:47)
   [junit4]   2> 
   [junit4]   2> "Signal Dispatcher" ID=4 RUNNABLE
   [junit4]   2> 
   [junit4]   2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@6d380f78
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@6d380f78
   [junit4]   2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
   [junit4]   2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
   [junit4]   2> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
   [junit4]   2> 
   [junit4]   2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@7b7fc74c
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on java.lang.ref.Reference$Lock@7b7fc74c
   [junit4]   2> 	at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
   [junit4]   2> 
   [junit4]   2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$1@58385dae
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$1@58385dae
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1281)
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1355)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:569)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:435)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:199)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:310)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
   [junit4]   2> 
   [junit4]   2> ^^==============================================
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDemoParallelLeafReader -Dtests.method=testRandomMultipleSchemaGensSameField -Dtests.seed=1A30B36FF7E830A0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_BH -Dtests.timezone=Etc/GMT-1 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   7200s J2 | TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)
   [junit4]   2> ??? 01, 2014 10:21:13 ? com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> ??? 01, 2014 10:21:34 ? com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 2 threads leaked from SUITE scope at org.apache.lucene.index.TestDemoParallelLeafReader: 
   [junit4]   2>    1) Thread[id=28931, name=Lucene Merge Thread #351, state=RUNNABLE, group=TGRP-TestDemoParallelLeafReader]
   [junit4]   2>         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
   [junit4]   2>         at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
   [junit4]   2>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
   [junit4]   2>         at sun.nio.ch.IOUtil.write(IOUtil.java:65)
   [junit4]   2>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
   [junit4]   2>         at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
   [junit4]   2>         at java.nio.channels.Channels.writeFully(Channels.java:101)
   [junit4]   2>         at java.nio.channels.Channels.access$000(Channels.java:61)
   [junit4]   2>         at java.nio.channels.Channels$1.write(Channels.java:174)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
   [junit4]   2>         at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:291)
   [junit4]   2>         at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
   [junit4]   2>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4]   2>         at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
   [junit4]   2>         at org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
   [junit4]   2>         at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:71)
   [junit4]   2>         at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:125)
   [junit4]   2>         at org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:111)
   [junit4]   2>         at org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:74)
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.addNumericFieldValues(DirectDocValuesConsumer.java:133)
   [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.addNumericField(DirectDocValuesConsumer.java:71)
   [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:110)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.mergeNumericField(DocValuesConsumer.java:252)
   [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:163)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:158)
   [junit4]   2>         at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:118)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3955)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   2>         at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:417)
   [junit4]   2>         at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:494)
   [junit4]   2>    2) Thread[id=22, name=TEST-TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField-seed#[1A30B36FF7E830A0], state=RUNNABLE, group=TGRP-TestDemoParallelLeafReader]
   [junit4]   2>         at sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
   [junit4]   2>         at sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:71)
   [junit4]   2>         at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
   [junit4]   2>         at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:136)
   [junit4]   2>         at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:149)
   [junit4]   2>         at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:175)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
   [junit4]   2>         at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:46)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
   [junit4]   2>         at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
   [junit4]   2>         at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
   [junit4]   2>         at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
   [junit4]   2>         at java.nio.channels.FileChannel.open(FileChannel.java:287)
   [junit4]   2>         at java.nio.channels.FileChannel.open(FileChannel.java:334)
   [junit4]   2>         at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:205)
   [junit4]   2>         at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
   [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:654)
   [junit4]   2>         at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
   [junit4]   2>         at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:91)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4504)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:509)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:476)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:514)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:379)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1398)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader$3.reindex(TestDemoParallelLeafReader.java:823)
   [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader.getParallelLeafReader(TestDemoParallelLeafReader.java:394)
   [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$1.warm(TestDemoParallelLeafReader.java:133)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4097)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   2>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> ??? 01, 2014 10:21:34 ? com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=28931, name=Lucene Merge Thread #351, state=BLOCKED, group=TGRP-TestDemoParallelLeafReader]
   [junit4]   2>    2) Thread[id=22, name=TEST-TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField-seed#[1A30B36FF7E830A0], state=RUNNABLE, group=TGRP-TestDemoParallelLeafReader]
   [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cq_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b))
   [junit4]   1> java.nio.channels.ClosedByInterruptException
   [junit4]   1> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
   [junit4]   1> 	at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
   [junit4]   1> 	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]   1> 	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.crash(MockDirectoryWrapper.java:337)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:762)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
   [junit4]   1> 	at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cz_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b; maxCacheMB=0.552734375 maxMergeSizeMB=0.47265625))
   [junit4]   1> java.nio.channels.ClosedByInterruptException
   [junit4]   1> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
   [junit4]   1> 	at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:208)
   [junit4]   1> 	at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:190)
   [junit4]   1> 	at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
   [junit4]   1> 	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
   [junit4]   1> 	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
   [junit4]   1> 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
   [junit4]   1> 	at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
   [junit4]   1> 	at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cx_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b)
   [junit4]   1> java.nio.channels.ClosedByInterruptException
   [junit4]   1> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
   [junit4]   1> 	at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
   [junit4]   1> 	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]   1> 	at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
   [junit4]   1> 	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
   [junit4]   1> 	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
   [junit4]   1> 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
   [junit4]   1> 	at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
   [junit4]   1> 	at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(SimpleFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9d2_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b))
   [junit4]   1> java.nio.channels.ClosedByInterruptException
   [junit4]   1> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
   [junit4]   1> 	at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterSeekableByteChannel.size(FilterSeekableByteChannel.java:81)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterSeekableByteChannel.size(FilterSeekableByteChannel.java:81)
   [junit4]   1> 	at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:89)
   [junit4]   1> 	at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:62)
   [junit4]   1> 	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
   [junit4]   1> 	at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
   [junit4]   1> 	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
   [junit4]   1> 	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
   [junit4]   1> 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
   [junit4]   1> 	at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
   [junit4]   1> 	at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9ct_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b)
   [junit4]   1> java.nio.channels.ClosedByInterruptException
   [junit4]   1> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
   [junit4]   1> 	at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:208)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.crash(MockDirectoryWrapper.java:337)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:762)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
   [junit4]   1> 	at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NRTCachingDirectory(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cw_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b; maxCacheMB=0.7490234375 maxMergeSizeMB=0.544921875))
   [junit4]   1> java.nio.channels.ClosedByInterruptException
   [junit4]   1> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
   [junit4]   1> 	at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
   [junit4]   1> 	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]   1> 	at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:190)
   [junit4]   1> 	at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
   [junit4]   1> 	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
   [junit4]   1> 	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
   [junit4]   1> 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
   [junit4]   1> 	at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
   [junit4]   1> 	at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cv_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b)
   [junit4]   1> java.nio.channels.ClosedByInterruptException
   [junit4]   1> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
   [junit4]   1> 	at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
   [junit4]   1> 	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]   1> 	at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
   [junit4]   1> 	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
   [junit4]   1> 	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
   [junit4]   1> 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
   [junit4]   1> 	at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
   [junit4]   1> 	at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(NRTCachingDirectory(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cs_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b; maxCacheMB=0.1884765625 maxMergeSizeMB=0.6328125)))
   [junit4]   1> java.nio.channels.ClosedByInterruptException
   [junit4]   1> 	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
   [junit4]   1> 	at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
   [junit4]   1> 	at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
   [junit4]   1> 	at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
   [junit4]   1> 	at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:190)
   [junit4]   1> 	at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.crash(MockDirectoryWrapper.java:346)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:762)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
   [junit4]   1> 	at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Statem

[...truncated too long message...]

rapper.java:644)
   [junit4]   1> 	at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
   [junit4]   1> 	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
   [junit4]   1> 	at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
   [junit4]   1> 	at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
   [junit4]   1> 	at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
   [junit4]   1> 	at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
   [junit4]   1> 	at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
   [junit4]   1> 	at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001
   [junit4]   2> Dec 01, 2014 9:21:34 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {number=PostingsFormat(name=LuceneVarGapDocFreqInterval), text=PostingsFormat(name=MockRandom), id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{number=DocValuesFormat(name=Direct)}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {text=IB LL-L3(800.0)}, locale=ar_BH, timezone=Etc/GMT-1
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=41795584,total=512229376
   [junit4]   2> NOTE: All tests run in this JVM: [TestTermsEnum2, TestDemoParallelLeafReader]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDemoParallelLeafReader -Dtests.seed=1A30B36FF7E830A0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_BH -Dtests.timezone=Etc/GMT-1 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J2 | TestDemoParallelLeafReader (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)
   [junit4] Completed on J2 in 7221.42s, 1 test, 2 errors <<< FAILURES!

[...truncated 195 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:536: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:477: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:49: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1363: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:970: There were test failures: 406 suites, 3026 tests, 1 suite-level error, 1 error, 37 ignored (27 assumptions)

Total time: 154 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #689
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 3591000 bytes
Compression is 0.0%
Took 1.3 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 690 - Failure

Posted by Michael McCandless <lu...@mikemccandless.com>.
I committed a fix: the test got MockRandomMP, causing O(N^2) merging....

Mike McCandless

http://blog.mikemccandless.com


On Mon, Dec 1, 2014 at 5:14 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> I'll dig.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, Dec 1, 2014 at 5:12 PM, Apache Jenkins Server
> <je...@builds.apache.org> wrote:
>> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/690/
>>
>> 2 tests failed.
>> REGRESSION:  org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField
>>
>> Error Message:
>> Test abandoned because suite timeout was reached.
>>
>> Stack Trace:
>> java.lang.Exception: Test abandoned because suite timeout was reached.
>>         at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)
>>
>>
>> FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestDemoParallelLeafReader
>>
>> Error Message:
>> Suite timeout exceeded (>= 7200000 msec).
>>
>> Stack Trace:
>> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>>         at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 1679 lines...]
>>    [junit4] Suite: org.apache.lucene.index.TestDemoParallelLeafReader
>>    [junit4]   2> ??? 01, 2014 10:21:13 ? com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
>>    [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestDemoParallelLeafReader
>>    [junit4]   2> ==== jstack at approximately timeout time ====
>>    [junit4]   2> "TEST-TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField-seed#[1A30B36FF7E830A0]" ID=22 RUNNABLE
>>    [junit4]   2>        at sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
>>    [junit4]   2>        at sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:71)
>>    [junit4]   2>        at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
>>    [junit4]   2>        at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:136)
>>    [junit4]   2>        at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:149)
>>    [junit4]   2>        at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:175)
>>    [junit4]   2>        at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>>    [junit4]   2>        at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:46)
>>    [junit4]   2>        at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>>    [junit4]   2>        at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
>>    [junit4]   2>        at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>>    [junit4]   2>        at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
>>    [junit4]   2>        at java.nio.channels.FileChannel.open(FileChannel.java:287)
>>    [junit4]   2>        at java.nio.channels.FileChannel.open(FileChannel.java:334)
>>    [junit4]   2>        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:205)
>>    [junit4]   2>        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>>    [junit4]   2>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:654)
>>    [junit4]   2>        - locked org.apache.lucene.store.MockDirectoryWrapper@3f04f430
>>    [junit4]   2>        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>>    [junit4]   2>        at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:91)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4504)
>>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:509)
>>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:476)
>>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:514)
>>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:379)
>>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1398)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   2>        at org.apache.lucene.index.TestDemoParallelLeafReader$3.reindex(TestDemoParallelLeafReader.java:823)
>>    [junit4]   2>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader.getParallelLeafReader(TestDemoParallelLeafReader.java:394)
>>    [junit4]   2>        - locked org.apache.lucene.index.TestDemoParallelLeafReader$3@1b66d903
>>    [junit4]   2>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$1.warm(TestDemoParallelLeafReader.java:133)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4097)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   2>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   2>        - locked org.apache.lucene.index.SerialMergeScheduler@3519b184
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   2>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   2>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   2>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   2>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   2>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   2>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   2>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   2>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   2>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>    [junit4]   2>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   2>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   2>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   2>
>>    [junit4]   2> "SUITE-TestDemoParallelLeafReader-seed#[1A30B36FF7E830A0]" ID=21 RUNNABLE
>>    [junit4]   2>        at sun.management.ThreadImpl.dumpThreads0(Native Method)
>>    [junit4]   2>        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:671)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$900(ThreadLeakControl.java:62)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:412)
>>    [junit4]   2>        - locked java.lang.Object@29d56c58
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:642)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:129)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:559)
>>    [junit4]   2>
>>    [junit4]   2> "JUnit4-serializer-daemon" ID=9 TIMED_WAITING
>>    [junit4]   2>        at java.lang.Thread.sleep(Native Method)
>>    [junit4]   2>        at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:47)
>>    [junit4]   2>
>>    [junit4]   2> "Signal Dispatcher" ID=4 RUNNABLE
>>    [junit4]   2>
>>    [junit4]   2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@6d380f78
>>    [junit4]   2>        at java.lang.Object.wait(Native Method)
>>    [junit4]   2>        - waiting on java.lang.ref.ReferenceQueue$Lock@6d380f78
>>    [junit4]   2>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>>    [junit4]   2>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>>    [junit4]   2>        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>>    [junit4]   2>
>>    [junit4]   2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@7b7fc74c
>>    [junit4]   2>        at java.lang.Object.wait(Native Method)
>>    [junit4]   2>        - waiting on java.lang.ref.Reference$Lock@7b7fc74c
>>    [junit4]   2>        at java.lang.Object.wait(Object.java:503)
>>    [junit4]   2>        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>>    [junit4]   2>
>>    [junit4]   2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$1@58385dae
>>    [junit4]   2>        at java.lang.Object.wait(Native Method)
>>    [junit4]   2>        - waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$1@58385dae
>>    [junit4]   2>        at java.lang.Thread.join(Thread.java:1281)
>>    [junit4]   2>        at java.lang.Thread.join(Thread.java:1355)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:569)
>>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:435)
>>    [junit4]   2>        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:199)
>>    [junit4]   2>        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:310)
>>    [junit4]   2>        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
>>    [junit4]   2>
>>    [junit4]   2> ^^==============================================
>>    [junit4]   2>
>>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDemoParallelLeafReader -Dtests.method=testRandomMultipleSchemaGensSameField -Dtests.seed=1A30B36FF7E830A0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_BH -Dtests.timezone=Etc/GMT-1 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>>    [junit4] ERROR   7200s J2 | TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField <<<
>>    [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
>>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)
>>    [junit4]   2> ??? 01, 2014 10:21:13 ? com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>>    [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
>>    [junit4]   2> ??? 01, 2014 10:21:34 ? com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>>    [junit4]   2> SEVERE: 2 threads leaked from SUITE scope at org.apache.lucene.index.TestDemoParallelLeafReader:
>>    [junit4]   2>    1) Thread[id=28931, name=Lucene Merge Thread #351, state=RUNNABLE, group=TGRP-TestDemoParallelLeafReader]
>>    [junit4]   2>         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>>    [junit4]   2>         at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>>    [junit4]   2>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>>    [junit4]   2>         at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>>    [junit4]   2>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>>    [junit4]   2>         at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>>    [junit4]   2>         at java.nio.channels.Channels.writeFully(Channels.java:101)
>>    [junit4]   2>         at java.nio.channels.Channels.access$000(Channels.java:61)
>>    [junit4]   2>         at java.nio.channels.Channels$1.write(Channels.java:174)
>>    [junit4]   2>         at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
>>    [junit4]   2>         at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
>>    [junit4]   2>         at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:291)
>>    [junit4]   2>         at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
>>    [junit4]   2>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>    [junit4]   2>         at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>>    [junit4]   2>         at org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
>>    [junit4]   2>         at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:71)
>>    [junit4]   2>         at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:125)
>>    [junit4]   2>         at org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:111)
>>    [junit4]   2>         at org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:74)
>>    [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.addNumericFieldValues(DirectDocValuesConsumer.java:133)
>>    [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.addNumericField(DirectDocValuesConsumer.java:71)
>>    [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:110)
>>    [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.mergeNumericField(DocValuesConsumer.java:252)
>>    [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:163)
>>    [junit4]   2>         at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:158)
>>    [junit4]   2>         at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:118)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3955)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   2>         at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:417)
>>    [junit4]   2>         at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:494)
>>    [junit4]   2>    2) Thread[id=22, name=TEST-TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField-seed#[1A30B36FF7E830A0], state=RUNNABLE, group=TGRP-TestDemoParallelLeafReader]
>>    [junit4]   2>         at sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
>>    [junit4]   2>         at sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:71)
>>    [junit4]   2>         at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
>>    [junit4]   2>         at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:136)
>>    [junit4]   2>         at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:149)
>>    [junit4]   2>         at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:175)
>>    [junit4]   2>         at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>>    [junit4]   2>         at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:46)
>>    [junit4]   2>         at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>>    [junit4]   2>         at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
>>    [junit4]   2>         at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>>    [junit4]   2>         at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
>>    [junit4]   2>         at java.nio.channels.FileChannel.open(FileChannel.java:287)
>>    [junit4]   2>         at java.nio.channels.FileChannel.open(FileChannel.java:334)
>>    [junit4]   2>         at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:205)
>>    [junit4]   2>         at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>>    [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:654)
>>    [junit4]   2>         at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>>    [junit4]   2>         at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:91)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4504)
>>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:509)
>>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:476)
>>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:514)
>>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:379)
>>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1398)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader$3.reindex(TestDemoParallelLeafReader.java:823)
>>    [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader.getParallelLeafReader(TestDemoParallelLeafReader.java:394)
>>    [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$1.warm(TestDemoParallelLeafReader.java:133)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4097)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   2>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>    [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   2> ??? 01, 2014 10:21:34 ? com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>>    [junit4]   2> INFO: Starting to interrupt leaked threads:
>>    [junit4]   2>    1) Thread[id=28931, name=Lucene Merge Thread #351, state=BLOCKED, group=TGRP-TestDemoParallelLeafReader]
>>    [junit4]   2>    2) Thread[id=22, name=TEST-TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField-seed#[1A30B36FF7E830A0], state=RUNNABLE, group=TGRP-TestDemoParallelLeafReader]
>>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cq_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b))
>>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
>>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
>>    [junit4]   1>        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.crash(MockDirectoryWrapper.java:337)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:762)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cz_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b; maxCacheMB=0.552734375 maxMergeSizeMB=0.47265625))
>>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:208)
>>    [junit4]   1>        at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:190)
>>    [junit4]   1>        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
>>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cx_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b)
>>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
>>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
>>    [junit4]   1>        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
>>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(SimpleFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9d2_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b))
>>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterSeekableByteChannel.size(FilterSeekableByteChannel.java:81)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterSeekableByteChannel.size(FilterSeekableByteChannel.java:81)
>>    [junit4]   1>        at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:89)
>>    [junit4]   1>        at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:62)
>>    [junit4]   1>        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>>    [junit4]   1>        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
>>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9ct_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b)
>>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:208)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.crash(MockDirectoryWrapper.java:337)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:762)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NRTCachingDirectory(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cw_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b; maxCacheMB=0.7490234375 maxMergeSizeMB=0.544921875))
>>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
>>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
>>    [junit4]   1>        at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:190)
>>    [junit4]   1>        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
>>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cv_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b)
>>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
>>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
>>    [junit4]   1>        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
>>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(NRTCachingDirectory(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cs_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b; maxCacheMB=0.1884765625 maxMergeSizeMB=0.6328125)))
>>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
>>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
>>    [junit4]   1>        at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:190)
>>    [junit4]   1>        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.crash(MockDirectoryWrapper.java:346)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:762)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Statem
>>
>> [...truncated too long message...]
>>
>> rapper.java:644)
>>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>>    [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001
>>    [junit4]   2> Dec 01, 2014 9:21:34 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>>    [junit4]   2> INFO: All leaked threads terminated.
>>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {number=PostingsFormat(name=LuceneVarGapDocFreqInterval), text=PostingsFormat(name=MockRandom), id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{number=DocValuesFormat(name=Direct)}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {text=IB LL-L3(800.0)}, locale=ar_BH, timezone=Etc/GMT-1
>>    [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=41795584,total=512229376
>>    [junit4]   2> NOTE: All tests run in this JVM: [TestTermsEnum2, TestDemoParallelLeafReader]
>>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDemoParallelLeafReader -Dtests.seed=1A30B36FF7E830A0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_BH -Dtests.timezone=Etc/GMT-1 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>>    [junit4] ERROR   0.00s J2 | TestDemoParallelLeafReader (suite) <<<
>>    [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)
>>    [junit4] Completed on J2 in 7221.42s, 1 test, 2 errors <<< FAILURES!
>>
>> [...truncated 195 lines...]
>> BUILD FAILED
>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:536: The following error occurred while executing this line:
>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:477: The following error occurred while executing this line:
>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:61: The following error occurred while executing this line:
>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: The following error occurred while executing this line:
>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:49: The following error occurred while executing this line:
>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1363: The following error occurred while executing this line:
>> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:970: There were test failures: 406 suites, 3026 tests, 1 suite-level error, 1 error, 37 ignored (27 assumptions)
>>
>> Total time: 154 minutes 5 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #689
>> Archived 5 artifacts
>> Archive block size is 32768
>> Received 0 blocks and 3591000 bytes
>> Compression is 0.0%
>> Took 1.3 sec
>> Recording test results
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Re: [JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 690 - Failure

Posted by Michael McCandless <lu...@mikemccandless.com>.
I'll dig.

Mike McCandless

http://blog.mikemccandless.com


On Mon, Dec 1, 2014 at 5:12 PM, Apache Jenkins Server
<je...@builds.apache.org> wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/690/
>
> 2 tests failed.
> REGRESSION:  org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField
>
> Error Message:
> Test abandoned because suite timeout was reached.
>
> Stack Trace:
> java.lang.Exception: Test abandoned because suite timeout was reached.
>         at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)
>
>
> FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestDemoParallelLeafReader
>
> Error Message:
> Suite timeout exceeded (>= 7200000 msec).
>
> Stack Trace:
> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>         at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)
>
>
>
>
> Build Log:
> [...truncated 1679 lines...]
>    [junit4] Suite: org.apache.lucene.index.TestDemoParallelLeafReader
>    [junit4]   2> ??? 01, 2014 10:21:13 ? com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
>    [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestDemoParallelLeafReader
>    [junit4]   2> ==== jstack at approximately timeout time ====
>    [junit4]   2> "TEST-TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField-seed#[1A30B36FF7E830A0]" ID=22 RUNNABLE
>    [junit4]   2>        at sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
>    [junit4]   2>        at sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:71)
>    [junit4]   2>        at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
>    [junit4]   2>        at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:136)
>    [junit4]   2>        at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:149)
>    [junit4]   2>        at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:175)
>    [junit4]   2>        at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>    [junit4]   2>        at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:46)
>    [junit4]   2>        at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>    [junit4]   2>        at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
>    [junit4]   2>        at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>    [junit4]   2>        at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
>    [junit4]   2>        at java.nio.channels.FileChannel.open(FileChannel.java:287)
>    [junit4]   2>        at java.nio.channels.FileChannel.open(FileChannel.java:334)
>    [junit4]   2>        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:205)
>    [junit4]   2>        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>    [junit4]   2>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:654)
>    [junit4]   2>        - locked org.apache.lucene.store.MockDirectoryWrapper@3f04f430
>    [junit4]   2>        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>    [junit4]   2>        at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:91)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4504)
>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:509)
>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:476)
>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:514)
>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:379)
>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1398)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   2>        at org.apache.lucene.index.TestDemoParallelLeafReader$3.reindex(TestDemoParallelLeafReader.java:823)
>    [junit4]   2>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader.getParallelLeafReader(TestDemoParallelLeafReader.java:394)
>    [junit4]   2>        - locked org.apache.lucene.index.TestDemoParallelLeafReader$3@1b66d903
>    [junit4]   2>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$1.warm(TestDemoParallelLeafReader.java:133)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4097)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   2>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   2>        - locked org.apache.lucene.index.SerialMergeScheduler@3519b184
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   2>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   2>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   2>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   2>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   2>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   2>        at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   2>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   2>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   2>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   2>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   2>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   2>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   2>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>    [junit4]   2>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   2>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   2>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   2>        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2>
>    [junit4]   2> "SUITE-TestDemoParallelLeafReader-seed#[1A30B36FF7E830A0]" ID=21 RUNNABLE
>    [junit4]   2>        at sun.management.ThreadImpl.dumpThreads0(Native Method)
>    [junit4]   2>        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:671)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$900(ThreadLeakControl.java:62)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:412)
>    [junit4]   2>        - locked java.lang.Object@29d56c58
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:642)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:129)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:559)
>    [junit4]   2>
>    [junit4]   2> "JUnit4-serializer-daemon" ID=9 TIMED_WAITING
>    [junit4]   2>        at java.lang.Thread.sleep(Native Method)
>    [junit4]   2>        at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:47)
>    [junit4]   2>
>    [junit4]   2> "Signal Dispatcher" ID=4 RUNNABLE
>    [junit4]   2>
>    [junit4]   2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@6d380f78
>    [junit4]   2>        at java.lang.Object.wait(Native Method)
>    [junit4]   2>        - waiting on java.lang.ref.ReferenceQueue$Lock@6d380f78
>    [junit4]   2>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
>    [junit4]   2>        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
>    [junit4]   2>        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
>    [junit4]   2>
>    [junit4]   2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@7b7fc74c
>    [junit4]   2>        at java.lang.Object.wait(Native Method)
>    [junit4]   2>        - waiting on java.lang.ref.Reference$Lock@7b7fc74c
>    [junit4]   2>        at java.lang.Object.wait(Object.java:503)
>    [junit4]   2>        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
>    [junit4]   2>
>    [junit4]   2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$1@58385dae
>    [junit4]   2>        at java.lang.Object.wait(Native Method)
>    [junit4]   2>        - waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$1@58385dae
>    [junit4]   2>        at java.lang.Thread.join(Thread.java:1281)
>    [junit4]   2>        at java.lang.Thread.join(Thread.java:1355)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:569)
>    [junit4]   2>        at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:435)
>    [junit4]   2>        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:199)
>    [junit4]   2>        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:310)
>    [junit4]   2>        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
>    [junit4]   2>
>    [junit4]   2> ^^==============================================
>    [junit4]   2>
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDemoParallelLeafReader -Dtests.method=testRandomMultipleSchemaGensSameField -Dtests.seed=1A30B36FF7E830A0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_BH -Dtests.timezone=Etc/GMT-1 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>    [junit4] ERROR   7200s J2 | TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField <<<
>    [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)
>    [junit4]   2> ??? 01, 2014 10:21:13 ? com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>    [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
>    [junit4]   2> ??? 01, 2014 10:21:34 ? com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>    [junit4]   2> SEVERE: 2 threads leaked from SUITE scope at org.apache.lucene.index.TestDemoParallelLeafReader:
>    [junit4]   2>    1) Thread[id=28931, name=Lucene Merge Thread #351, state=RUNNABLE, group=TGRP-TestDemoParallelLeafReader]
>    [junit4]   2>         at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>    [junit4]   2>         at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
>    [junit4]   2>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
>    [junit4]   2>         at sun.nio.ch.IOUtil.write(IOUtil.java:65)
>    [junit4]   2>         at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
>    [junit4]   2>         at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
>    [junit4]   2>         at java.nio.channels.Channels.writeFully(Channels.java:101)
>    [junit4]   2>         at java.nio.channels.Channels.access$000(Channels.java:61)
>    [junit4]   2>         at java.nio.channels.Channels$1.write(Channels.java:174)
>    [junit4]   2>         at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
>    [junit4]   2>         at org.apache.lucene.mockfile.FilterOutputStream2.write(FilterOutputStream2.java:63)
>    [junit4]   2>         at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:291)
>    [junit4]   2>         at java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
>    [junit4]   2>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>    [junit4]   2>         at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
>    [junit4]   2>         at org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
>    [junit4]   2>         at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:71)
>    [junit4]   2>         at org.apache.lucene.store.MockIndexOutputWrapper.writeBytes(MockIndexOutputWrapper.java:125)
>    [junit4]   2>         at org.apache.lucene.store.MockIndexOutputWrapper.writeByte(MockIndexOutputWrapper.java:111)
>    [junit4]   2>         at org.apache.lucene.store.DataOutput.writeInt(DataOutput.java:74)
>    [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.addNumericFieldValues(DirectDocValuesConsumer.java:133)
>    [junit4]   2>         at org.apache.lucene.codecs.memory.DirectDocValuesConsumer.addNumericField(DirectDocValuesConsumer.java:71)
>    [junit4]   2>         at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:110)
>    [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.mergeNumericField(DocValuesConsumer.java:252)
>    [junit4]   2>         at org.apache.lucene.codecs.DocValuesConsumer.merge(DocValuesConsumer.java:163)
>    [junit4]   2>         at org.apache.lucene.index.SegmentMerger.mergeDocValues(SegmentMerger.java:158)
>    [junit4]   2>         at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:118)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3955)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   2>         at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:417)
>    [junit4]   2>         at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:494)
>    [junit4]   2>    2) Thread[id=22, name=TEST-TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField-seed#[1A30B36FF7E830A0], state=RUNNABLE, group=TGRP-TestDemoParallelLeafReader]
>    [junit4]   2>         at sun.nio.fs.UnixNativeDispatcher.open0(Native Method)
>    [junit4]   2>         at sun.nio.fs.UnixNativeDispatcher.open(UnixNativeDispatcher.java:71)
>    [junit4]   2>         at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:258)
>    [junit4]   2>         at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:136)
>    [junit4]   2>         at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:149)
>    [junit4]   2>         at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:175)
>    [junit4]   2>         at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>    [junit4]   2>         at org.apache.lucene.mockfile.DisableFsyncFS.newFileChannel(DisableFsyncFS.java:46)
>    [junit4]   2>         at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>    [junit4]   2>         at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
>    [junit4]   2>         at org.apache.lucene.mockfile.FilterFileSystemProvider.newFileChannel(FilterFileSystemProvider.java:180)
>    [junit4]   2>         at org.apache.lucene.mockfile.HandleTrackingFS.newFileChannel(HandleTrackingFS.java:151)
>    [junit4]   2>         at java.nio.channels.FileChannel.open(FileChannel.java:287)
>    [junit4]   2>         at java.nio.channels.FileChannel.open(FileChannel.java:334)
>    [junit4]   2>         at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:205)
>    [junit4]   2>         at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>    [junit4]   2>         at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:654)
>    [junit4]   2>         at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>    [junit4]   2>         at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:91)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:4504)
>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:509)
>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:476)
>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:514)
>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:379)
>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:478)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1398)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader$3.reindex(TestDemoParallelLeafReader.java:823)
>    [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader.getParallelLeafReader(TestDemoParallelLeafReader.java:394)
>    [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$1.warm(TestDemoParallelLeafReader.java:133)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4097)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   2>         at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   2>         at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> ??? 01, 2014 10:21:34 ? com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>    [junit4]   2> INFO: Starting to interrupt leaked threads:
>    [junit4]   2>    1) Thread[id=28931, name=Lucene Merge Thread #351, state=BLOCKED, group=TGRP-TestDemoParallelLeafReader]
>    [junit4]   2>    2) Thread[id=22, name=TEST-TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField-seed#[1A30B36FF7E830A0], state=RUNNABLE, group=TGRP-TestDemoParallelLeafReader]
>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cq_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b))
>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
>    [junit4]   1>        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.crash(MockDirectoryWrapper.java:337)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:762)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NRTCachingDirectory(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cz_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b; maxCacheMB=0.552734375 maxMergeSizeMB=0.47265625))
>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:208)
>    [junit4]   1>        at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:190)
>    [junit4]   1>        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cx_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b)
>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
>    [junit4]   1>        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(SimpleFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9d2_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b))
>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterSeekableByteChannel.size(FilterSeekableByteChannel.java:81)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterSeekableByteChannel.size(FilterSeekableByteChannel.java:81)
>    [junit4]   1>        at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:89)
>    [junit4]   1>        at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:62)
>    [junit4]   1>        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>    [junit4]   1>        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(MMapDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9ct_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b)
>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:208)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.crash(MockDirectoryWrapper.java:337)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:762)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NRTCachingDirectory(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cw_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b; maxCacheMB=0.7490234375 maxMergeSizeMB=0.544921875))
>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
>    [junit4]   1>        at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:190)
>    [junit4]   1>        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cv_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b)
>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
>    [junit4]   1>        at org.apache.lucene.util.LuceneTestCase.slowFileExists(LuceneTestCase.java:2473)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:644)
>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   1> TEST: hit IOExc closing dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(NRTCachingDirectory(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001/tempDir-001/segs/akmlsnwfygmohvapz2cqsa9cs_89 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5cfd0d6b; maxCacheMB=0.1884765625 maxMergeSizeMB=0.6328125)))
>    [junit4]   1> java.nio.channels.ClosedByInterruptException
>    [junit4]   1>        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
>    [junit4]   1>        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:312)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.mockfile.FilterFileChannel.size(FilterFileChannel.java:88)
>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:108)
>    [junit4]   1>        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:81)
>    [junit4]   1>        at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:190)
>    [junit4]   1>        at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.crash(MockDirectoryWrapper.java:346)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:762)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Statem
>
> [...truncated too long message...]
>
> rapper.java:644)
>    [junit4]   1>        at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
>    [junit4]   1>        at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:282)
>    [junit4]   1>        at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:464)
>    [junit4]   1>        at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:232)
>    [junit4]   1>        at org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:766)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ParallelReaderClosed.onClose(TestDemoParallelLeafReader.java:339)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.notifyReaderClosedListeners(IndexReader.java:145)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:260)
>    [junit4]   1>        at org.apache.lucene.index.ParallelLeafReader.doClose(ParallelLeafReader.java:257)
>    [junit4]   1>        at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:253)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader$ReindexingReader$ReindexingMergePolicy$ReindexingOneMerge.mergeFinished(TestDemoParallelLeafReader.java:545)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.closeMergeReaders(IndexWriter.java:3841)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4118)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3548)
>    [junit4]   1>        at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1861)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.doAfterSegmentFlushed(IndexWriter.java:4558)
>    [junit4]   1>        at org.apache.lucene.index.DocumentsWriter$MergePendingEvent.process(DocumentsWriter.java:706)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4583)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4575)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1399)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1133)
>    [junit4]   1>        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1118)
>    [junit4]   1>        at org.apache.lucene.index.TestDemoParallelLeafReader.testRandomMultipleSchemaGensSameField(TestDemoParallelLeafReader.java:1053)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   1>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>    [junit4]   1>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   1>        at java.lang.reflect.Method.invoke(Method.java:606)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>    [junit4]   1>        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>    [junit4]   1>        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   1>        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>    [junit4]   1>        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build/core/test/J2/temp/lucene.index.TestDemoParallelLeafReader-1A30B36FF7E830A0-001
>    [junit4]   2> Dec 01, 2014 9:21:34 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>    [junit4]   2> INFO: All leaked threads terminated.
>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {number=PostingsFormat(name=LuceneVarGapDocFreqInterval), text=PostingsFormat(name=MockRandom), id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{number=DocValuesFormat(name=Direct)}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {text=IB LL-L3(800.0)}, locale=ar_BH, timezone=Etc/GMT-1
>    [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=41795584,total=512229376
>    [junit4]   2> NOTE: All tests run in this JVM: [TestTermsEnum2, TestDemoParallelLeafReader]
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDemoParallelLeafReader -Dtests.seed=1A30B36FF7E830A0 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar_BH -Dtests.timezone=Etc/GMT-1 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>    [junit4] ERROR   0.00s J2 | TestDemoParallelLeafReader (suite) <<<
>    [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([1A30B36FF7E830A0]:0)
>    [junit4] Completed on J2 in 7221.42s, 1 test, 2 errors <<< FAILURES!
>
> [...truncated 195 lines...]
> BUILD FAILED
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:536: The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:477: The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:61: The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/build.xml:49: The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1363: The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:970: There were test failures: 406 suites, 3026 tests, 1 suite-level error, 1 error, 37 ignored (27 assumptions)
>
> Total time: 154 minutes 5 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #689
> Archived 5 artifacts
> Archive block size is 32768
> Received 0 blocks and 3591000 bytes
> Compression is 0.0%
> Took 1.3 sec
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 695 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/695/

3 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
The Monkey ran for over 20 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 20 seconds and no jetties were stopped - this is worth investigating!
	at __randomizedtesting.SeedInfo.seed([890D9572BE2EEE87:8EB1B6AC9718EBB]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:535)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:140)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:     1) Thread[id=4210, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)    2) Thread[id=4211, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest: 
   1) Thread[id=4210, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)
   2) Thread[id=4211, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)
	at __randomizedtesting.SeedInfo.seed([890D9572BE2EEE87]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=4210, name=StopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Class.getGenericSignature(Native Method)         at java.lang.Class.getGenericSuperclass(Class.java:690)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:120)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:65)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.MiniGson.getAdapter(MiniGson.java:92)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:504)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:87)         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:410)         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)         at java.io.PrintStream.flush(PrintStream.java:338)         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)         at java.io.PrintStream.write(PrintStream.java:482)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at java.io.PrintStream.newLine(PrintStream.java:546)         at java.io.PrintStream.println(PrintStream.java:824)         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at java.lang.Throwable.printStackTrace(Throwable.java:658)         at java.lang.Throwable.printStackTrace(Throwable.java:643)         at java.lang.Throwable.printStackTrace(Throwable.java:634)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:108)    2) Thread[id=4211, name=StopableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.io.PrintStream.println(PrintStream.java:806)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:107)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=4210, name=StopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Class.getGenericSignature(Native Method)
        at java.lang.Class.getGenericSuperclass(Class.java:690)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:120)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:65)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.MiniGson.getAdapter(MiniGson.java:92)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:504)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:87)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:410)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.PrintStream.write(PrintStream.java:482)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.newLine(PrintStream.java:546)
        at java.io.PrintStream.println(PrintStream.java:824)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:108)
   2) Thread[id=4211, name=StopableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.io.PrintStream.println(PrintStream.java:806)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:107)
	at __randomizedtesting.SeedInfo.seed([890D9572BE2EEE87]:0)




Build Log:
[...truncated 10143 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/init-core-data-001
   [junit4]   2> 1343854 T4051 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1343854 T4051 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1343861 T4051 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1343862 T4051 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1343863 T4052 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1343963 T4051 oasc.ZkTestServer.run start zk server on port:48999
   [junit4]   2> 1343964 T4051 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1343965 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1343971 T4059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@485c78e7 name:ZooKeeperConnection Watcher:127.0.0.1:48999 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1343971 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1343972 T4051 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1343972 T4051 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1343975 T4051 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1343976 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1343978 T4062 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d04c709 name:ZooKeeperConnection Watcher:127.0.0.1:48999/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1343978 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1343979 T4051 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1343979 T4051 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1343981 T4051 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1343983 T4051 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1343984 T4051 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1343987 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1343988 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1343991 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1343991 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1343994 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1343995 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1343997 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1343998 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1344000 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1344001 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1344003 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1344004 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1344006 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1344007 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1344009 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1344010 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1344012 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1344013 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1344015 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1344016 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1344018 T4051 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1344019 T4051 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1344021 T4051 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1344021 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1344023 T4065 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fea78f3 name:ZooKeeperConnection Watcher:127.0.0.1:48999/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1344023 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1344024 T4051 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1346308 T4051 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1346317 T4051 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1346328 T4051 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:31994
   [junit4]   2> 1346336 T4051 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1346337 T4051 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1346337 T4051 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-002
   [junit4]   2> 1346338 T4051 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-002/'
   [junit4]   2> 1346493 T4051 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-002/solr.xml
   [junit4]   2> 1346706 T4051 oasc.CoreContainer.<init> New CoreContainer 86515915
   [junit4]   2> 1346706 T4051 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-002/]
   [junit4]   2> 1346708 T4051 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1346708 T4051 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1346709 T4051 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1346709 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1346709 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1346710 T4051 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1346710 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1346710 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1346711 T4051 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1346711 T4051 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1346715 T4051 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1346718 T4051 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1346719 T4051 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1346719 T4051 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1346719 T4051 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48999/solr
   [junit4]   2> 1346720 T4051 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1346720 T4051 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1346721 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1346723 T4077 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22fa9d73 name:ZooKeeperConnection Watcher:127.0.0.1:48999 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1346725 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1346725 T4051 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1346728 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1346730 T4080 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55fc67a3 name:ZooKeeperConnection Watcher:127.0.0.1:48999/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1346730 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1346732 T4051 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1346735 T4051 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1346738 T4051 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1346741 T4051 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1346744 T4051 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1346748 T4051 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1346750 T4051 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31994_
   [junit4]   2> 1346751 T4051 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31994_
   [junit4]   2> 1346753 T4051 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1346755 T4051 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1346757 T4051 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1346760 T4051 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:31994_
   [junit4]   2> 1346760 T4051 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1346762 T4051 oasc.Overseer.start Overseer (id=92922865046650884-127.0.0.1:31994_-n_0000000000) starting
   [junit4]   2> 1346765 T4051 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1346774 T4051 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1346775 T4082 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1346776 T4051 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1346778 T4051 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1346779 T4051 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1346783 T4081 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1346797 T4084 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1346798 T4084 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1346799 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1346799 T4084 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1346802 T4081 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:31994",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:31994_",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1346802 T4081 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1346802 T4081 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 1346803 T4081 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1346806 T4080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1347799 T4084 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1347800 T4084 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1347800 T4084 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1347801 T4084 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1347801 T4084 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1347801 T4084 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-002/collection1/'
   [junit4]   2> 1347804 T4084 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1347805 T4084 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1347806 T4084 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1347855 T4084 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1347858 T4084 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1347899 T4084 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1347955 T4084 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1347956 T4084 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1347968 T4084 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1348420 T4084 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1348423 T4084 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1348425 T4084 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1348432 T4084 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1348453 T4084 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1348453 T4084 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1348453 T4084 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/control/data/
   [junit4]   2> 1348454 T4084 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c8873e3
   [junit4]   2> 1348463 T4084 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/control/data
   [junit4]   2> 1348467 T4084 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/control/data/index/
   [junit4]   2> 1348468 T4084 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1348469 T4084 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/control/data/index
   [junit4]   2> 1348470 T4084 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=41.9638671875, floorSegmentMB=0.619140625, forceMergeDeletesPctAllowed=3.2584979477639466, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1348490 T4084 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1348490 T4084 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1348495 T4084 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1348495 T4084 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1348495 T4084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1348496 T4084 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1348496 T4084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1348496 T4084 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1348497 T4084 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1348497 T4084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1348497 T4084 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1348498 T4084 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1348498 T4084 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1348499 T4084 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1348499 T4084 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1348500 T4084 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1348500 T4084 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1348500 T4084 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 1348501 T4084 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1348501 T4084 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1348502 T4084 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1348503 T4084 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1348503 T4084 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1348504 T4084 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1348504 T4084 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1348505 T4084 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1348505 T4084 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1348506 T4084 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1348506 T4084 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1348506 T4084 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1348537 T4084 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1348539 T4084 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1348542 T4084 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1348544 T4084 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1348549 T4084 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1348549 T4084 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1348550 T4084 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=0.6318359375, floorSegmentMB=1.9111328125, forceMergeDeletesPctAllowed=26.11671437287306, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1348552 T4084 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1348552 T4084 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1348552 T4084 oass.SolrIndexSearcher.<init> Opening Searcher@1e21185d[collection1] main
   [junit4]   2> 1348553 T4084 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1348553 T4084 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1348554 T4084 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1348554 T4084 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1348554 T4084 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1348555 T4084 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1348555 T4084 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1348555 T4084 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1348556 T4084 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 1348558 T4084 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1348558 T4084 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1348558 T4084 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1348558 T4084 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1348559 T4085 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e21185d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1348560 T4084 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1348561 T4088 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:31994 collection:control_collection shard:shard1
   [junit4]   2> 1348561 T4051 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0
   [junit4]   2> 1348561 T4051 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1348561 T4088 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1348566 T4088 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1348567 T4051 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1348568 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1348568 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1348568 T4088 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1348569 T4088 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C586 name=collection1 org.apache.solr.core.SolrCore@23f74a80 url=https://127.0.0.1:31994/collection1 node=127.0.0.1:31994_ C586_STATE=coll:control_collection core:collection1 props:{base_url=https://127.0.0.1:31994, state=down, node_name=127.0.0.1:31994_, core=collection1}
   [junit4]   2> 1348569 T4088 C586 P31994 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:31994/collection1/
   [junit4]   2> 1348569 T4088 C586 P31994 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1348569 T4088 C586 P31994 oasc.SyncStrategy.syncToMe https://127.0.0.1:31994/collection1/ has no replicas
   [junit4]   2> 1348570 T4088 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:31994/collection1/ shard1
   [junit4]   2> 1348570 T4091 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@672615d name:ZooKeeperConnection Watcher:127.0.0.1:48999/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1348570 T4088 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1348570 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1348571 T4051 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1348571 T4081 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"}
   [junit4]   2> 1348572 T4051 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1348574 T4080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1348574 T4091 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 0)
   [junit4]   2> 1348575 T4051 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1348576 T4051 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1348576 T4051 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1348577 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1348578 T4094 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d901b33 name:ZooKeeperConnection Watcher:127.0.0.1:48999/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1348579 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1348579 T4081 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:31994",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 1348579 T4051 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1348581 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1348582 T4080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1348582 T4091 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1348585 T4081 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"5",
   [junit4]   2> 	  "stateFormat":2}
   [junit4]   2> 1348586 T4081 oasc.Overseer$ClusterStateUpdater.buildCollection Building a new collection: collection1
   [junit4]   2> 1348586 T4081 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4, shard5]
   [junit4]   2> 1348586 T4081 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1348588 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1348588 T4081 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1348589 T4080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1348589 T4091 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1348625 T4088 oasc.ZkController.register We are https://127.0.0.1:31994/collection1/ and leader is https://127.0.0.1:31994/collection1/
   [junit4]   2> 1348626 T4088 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:31994
   [junit4]   2> 1348626 T4088 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1348626 T4088 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1348626 T4088 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1348628 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1348630 T4081 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"5",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:31994",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:31994_",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1348733 T4080 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1348733 T4091 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1350953 T4051 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1350954 T4051 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1350961 T4051 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1350962 T4051 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:38381
   [junit4]   2> 1350967 T4051 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1350967 T4051 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1350968 T4051 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-003
   [junit4]   2> 1350968 T4051 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-003/'
   [junit4]   2> 1351035 T4051 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-003/solr.xml
   [junit4]   2> 1351157 T4051 oasc.CoreContainer.<init> New CoreContainer 899446855
   [junit4]   2> 1351158 T4051 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-003/]
   [junit4]   2> 1351159 T4051 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1351160 T4051 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1351160 T4051 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1351160 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1351161 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1351161 T4051 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1351161 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1351162 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1351162 T4051 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1351162 T4051 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1351165 T4051 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1351168 T4051 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1351168 T4051 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1351169 T4051 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1351169 T4051 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48999/solr
   [junit4]   2> 1351169 T4051 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1351170 T4051 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1351171 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1351172 T4106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4081a130 name:ZooKeeperConnection Watcher:127.0.0.1:48999 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1351173 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1351174 T4051 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1351176 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1351177 T4109 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61933808 name:ZooKeeperConnection Watcher:127.0.0.1:48999/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1351178 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1351182 T4051 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1352187 T4051 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38381_
   [junit4]   2> 1352189 T4051 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38381_
   [junit4]   2> 1352192 T4051 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1352201 T4110 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1352201 T4110 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1352203 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1352203 T4110 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 1352203 T4110 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1352204 T4110 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1352205 T4110 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 1352205 T4110 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1352208 T4081 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38381",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38381_",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1352210 T4081 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=5
   [junit4]   2> 1352211 T4081 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1352214 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1352214 T4081 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1352215 T4109 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1352216 T4109 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 1353206 T4110 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1353207 T4110 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1353208 T4110 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1353208 T4110 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1353208 T4110 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1353209 T4110 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-003/collection1/'
   [junit4]   2> 1353211 T4110 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1353212 T4110 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1353213 T4110 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1353264 T4110 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1353267 T4110 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1353306 T4110 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1353363 T4110 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1353364 T4110 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1353375 T4110 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1353787 T4110 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1353790 T4110 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1353792 T4110 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1353797 T4110 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1353821 T4110 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1353822 T4110 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1353822 T4110 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty1/
   [junit4]   2> 1353822 T4110 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c8873e3
   [junit4]   2> 1353824 T4110 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty1
   [junit4]   2> 1353825 T4110 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty1/index/
   [junit4]   2> 1353826 T4110 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1353828 T4110 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty1/index
   [junit4]   2> 1353828 T4110 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=41.9638671875, floorSegmentMB=0.619140625, forceMergeDeletesPctAllowed=3.2584979477639466, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1353847 T4110 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1353848 T4110 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1353854 T4110 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1353854 T4110 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1353854 T4110 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1353855 T4110 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1353855 T4110 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1353855 T4110 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1353856 T4110 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1353856 T4110 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1353856 T4110 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1353857 T4110 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1353857 T4110 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1353858 T4110 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1353859 T4110 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1353859 T4110 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1353860 T4110 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1353862 T4110 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 1353863 T4110 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1353864 T4110 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1353865 T4110 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1353865 T4110 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1353866 T4110 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1353866 T4110 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1353867 T4110 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1353868 T4110 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1353868 T4110 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1353869 T4110 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1353869 T4110 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1353870 T4110 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1353901 T4110 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1353903 T4110 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1353906 T4110 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1353908 T4110 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1353913 T4110 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1353913 T4110 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1353915 T4110 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=0.6318359375, floorSegmentMB=1.9111328125, forceMergeDeletesPctAllowed=26.11671437287306, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1353916 T4110 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1353917 T4110 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1353917 T4110 oass.SolrIndexSearcher.<init> Opening Searcher@9f62e55[collection1] main
   [junit4]   2> 1353917 T4110 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1353918 T4110 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1353918 T4110 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1353919 T4110 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1353919 T4110 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1353919 T4110 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1353920 T4110 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1353920 T4110 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1353920 T4110 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1353921 T4110 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1353921 T4110 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1353922 T4111 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9f62e55[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1353922 T4110 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1353923 T4114 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:38381 collection:collection1 shard:shard2
   [junit4]   2> 1353923 T4051 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0
   [junit4]   2> 1353924 T4051 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1353924 T4114 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1353928 T4114 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1353929 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1353929 T4114 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1353929 T4114 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C587 name=collection1 org.apache.solr.core.SolrCore@4817864c url=https://127.0.0.1:38381/collection1 node=127.0.0.1:38381_ C587_STATE=coll:collection1 core:collection1 props:{base_url=https://127.0.0.1:38381, state=down, node_name=127.0.0.1:38381_, core=collection1}
   [junit4]   2> 1353930 T4114 C587 P38381 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:38381/collection1/
   [junit4]   2> 1353930 T4114 C587 P38381 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1353930 T4114 C587 P38381 oasc.SyncStrategy.syncToMe https://127.0.0.1:38381/collection1/ has no replicas
   [junit4]   2> 1353930 T4114 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:38381/collection1/ shard2
   [junit4]   2> 1353931 T4114 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1353931 T4081 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"}
   [junit4]   2> 1353934 T4081 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1353935 T4109 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1353935 T4109 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 1353938 T4081 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38381",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"}
   [junit4]   2> 1353940 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1353940 T4081 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1353941 T4109 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1353942 T4109 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 1353985 T4114 oasc.ZkController.register We are https://127.0.0.1:38381/collection1/ and leader is https://127.0.0.1:38381/collection1/
   [junit4]   2> 1353986 T4114 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:38381
   [junit4]   2> 1353986 T4114 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1353986 T4114 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1353987 T4114 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1353988 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1353991 T4114 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 1353991 T4081 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "numShards":"5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:38381",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:38381_",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1353996 T4081 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1353996 T4109 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1353997 T4109 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 1356496 T4051 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1356497 T4051 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1356504 T4051 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1356506 T4051 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:45839
   [junit4]   2> 1356509 T4051 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1356510 T4051 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1356510 T4051 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-004
   [junit4]   2> 1356511 T4051 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-004/'
   [junit4]   2> 1356573 T4051 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-004/solr.xml
   [junit4]   2> 1356697 T4051 oasc.CoreContainer.<init> New CoreContainer 2036719034
   [junit4]   2> 1356697 T4051 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-004/]
   [junit4]   2> 1356699 T4051 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1356699 T4051 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1356699 T4051 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1356700 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1356700 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1356700 T4051 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1356701 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1356701 T4051 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1356701 T4051 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1356702 T4051 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1356705 T4051 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1356707 T4051 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1356708 T4051 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1356708 T4051 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1356709 T4051 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48999/solr
   [junit4]   2> 1356709 T4051 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1356709 T4051 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1356710 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1356712 T4126 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@578b8a47 name:ZooKeeperConnection Watcher:127.0.0.1:48999 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1356713 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1356713 T4051 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1356715 T4051 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1356717 T4129 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b9a7951 name:ZooKeeperConnection Watcher:127.0.0.1:48999/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1356717 T4051 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1356722 T4051 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1357726 T4051 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45839_
   [junit4]   2> 1357727 T4051 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45839_
   [junit4]   2> 1357730 T4051 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1357740 T4130 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1357740 T4130 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1357742 T4080 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1357742 T4130 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 1357742 T4130 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1357743 T4130 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1357744 T4130 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 1357744 T4130 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1357744 T4081 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "numShards":"5",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:45839",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:45839_",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1357747 T4081 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=5
   [junit4]   2> 1357747 T4081 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard5
   [junit4]   2> 1357751 T4081 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1357752 T4109 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1357752 T4129 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1357752 T4109 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 1357753 T4129 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 1358744 T4130 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1358745 T4130 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1358746 T4130 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1358746 T4130 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1358747 T4130 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1358747 T4130 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-004/collection1/'
   [junit4]   2> 1358751 T4130 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 1358752 T4130 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 1358754 T4130 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 1358811 T4130 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1358814 T4130 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1358859 T4130 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1358924 T4130 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1358926 T4130 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1358937 T4130 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1359443 T4130 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1359447 T4130 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1359450 T4130 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1359456 T4130 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1359478 T4130 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1359478 T4130 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1359478 T4130 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty2/
   [junit4]   2> 1359479 T4130 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c8873e3
   [junit4]   2> 1359482 T4130 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty2
   [junit4]   2> 1359484 T4130 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty2/index/
   [junit4]   2> 1359487 T4130 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1359489 T4130 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty2/index
   [junit4]   2> 1359490 T4130 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=39, maxMergedSegmentMB=41.9638671875, floorSegmentMB=0.619140625, forceMergeDeletesPctAllowed=3.2584979477639466, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1359519 T4130 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-890D9572BE2EEE87-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1359519 T4130 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1359526 T4130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1359526 T4130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1359526 T4130 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1359527 T4130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1359527 T4130 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1359527 T4130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1359528 T4130 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1359528 T4130 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1359528 T4130 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1359529 T4130 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1359530 T4130 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1359530 T4130 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1359531 T4130 oasc.RequestHandlers.ini

[...truncated too long message...]

requestConnection(PoolingClientConnectionManager.java:184)
   [junit4]   2> 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:414)
   [junit4]   2> 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:465)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]   2> 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.indexDoc(StopableIndexingThread.java:168)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.indexr(StopableIndexingThread.java:158)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:103)
   [junit4]   2> java.lang.IllegalStateException: Connection pool shut down
   [junit4]   2> 	at org.apache.http.util.Asserts.check(Asserts.java:34)
   [junit4]   2> 	at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:169)
   [junit4]   2> 	at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:202)
   [junit4]   2> 	at org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:184)
   [junit4]   2> 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:414)
   [junit4]   2> 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]   2> 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:465)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]   2> 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.indexDoc(StopableIndexingThread.java:168)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.indexr(StopableIndexingThread.java:158)
   [junit4]   2> 	at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:103)
   [junit4]   2> 124074 T4211 ccr.RunnerThreadGroup.uncaughtException SEVERE RunnerThreadGroup's sub thread should always have a context and it didn't have any? java.lang.IllegalStateException: No context information for thread: Thread[id=4211, name=StopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]. Is this thread running under a class com.carrotsearch.randomizedtesting.RandomizedRunner runner context? Add @RunWith(class com.carrotsearch.randomizedtesting.RandomizedRunner.class) to your test class. Make sure your code accesses random contexts within @BeforeClass and @AfterClass boundary (for example, static test class initializers are not permitted to access random contexts).
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:206)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:144)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.augmentStackTrace(RandomizedRunner.java:1753)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RunnerThreadGroup.uncaughtException(RunnerThreadGroup.java:23)
   [junit4]   2> 	at java.lang.Thread.dispatchUncaughtException(Thread.java:1986)
   [junit4]   2> 
   [junit4] ERROR   0.00s J0 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest: 
   [junit4]    >    1) Thread[id=4210, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)
   [junit4]    >    2) Thread[id=4211, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([890D9572BE2EEE87]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=4210, name=StopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.lang.Class.getGenericSignature(Native Method)
   [junit4]    >         at java.lang.Class.getGenericSuperclass(Class.java:690)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:120)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:65)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.MiniGson.getAdapter(MiniGson.java:92)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:504)
   [junit4]    >         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:87)
   [junit4]    >         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
   [junit4]    >         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.flush(PrintStream.java:338)
   [junit4]    >         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
   [junit4]    >         at java.io.PrintStream.write(PrintStream.java:482)
   [junit4]    >         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
   [junit4]    >         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
   [junit4]    >         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
   [junit4]    >         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
   [junit4]    >         at java.io.PrintStream.newLine(PrintStream.java:546)
   [junit4]    >         at java.io.PrintStream.println(PrintStream.java:824)
   [junit4]    >         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
   [junit4]    >         at java.lang.Throwable.printStackTrace(Throwable.java:658)
   [junit4]    >         at java.lang.Throwable.printStackTrace(Throwable.java:643)
   [junit4]    >         at java.lang.Throwable.printStackTrace(Throwable.java:634)
   [junit4]    >         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:108)
   [junit4]    >    2) Thread[id=4211, name=StopableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    >         at java.io.PrintStream.println(PrintStream.java:806)
   [junit4]    >         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:107)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([890D9572BE2EEE87]:0)
   [junit4] Completed on J0 in 124.11s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 935 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:536: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:477: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:190: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:508: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1363: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:970: There were test failures: 450 suites, 1844 tests, 2 suite-level errors, 1 failure, 976 ignored (7 assumptions)

Total time: 153 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #689
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 335914814 bytes
Compression is 0.0%
Took 59 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 694 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/694/

2 tests failed.
REGRESSION:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([4711AD7639F6F7]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([4711AD7639F6F7]:0)




Build Log:
[...truncated 11034 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/init-core-data-001
   [junit4]   2> 1547885 T7619 oas.SolrTestCaseJ4.setUp ###Starting doTestDetails
   [junit4]   2> 1547902 T7619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1547913 T7619 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:43367
   [junit4]   2> 1547914 T7619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1547915 T7619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1547915 T7619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001
   [junit4]   2> 1547915 T7619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/'
   [junit4]   2> 1547963 T7619 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/solr.xml
   [junit4]   2> 1548072 T7619 oasc.CoreContainer.<init> New CoreContainer 231193030
   [junit4]   2> 1548073 T7619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/]
   [junit4]   2> 1548074 T7619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1548074 T7619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1548075 T7619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1548075 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1548075 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1548076 T7619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1548076 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1548076 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1548077 T7619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1548077 T7619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1548078 T7619 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1548078 T7619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1548079 T7619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1548079 T7619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1548084 T7629 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/'
   [junit4]   2> 1548139 T7629 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1548182 T7629 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1548183 T7629 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/conf/schema.xml
   [junit4]   2> 1548190 T7629 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1548207 T7629 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1548208 T7629 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/
   [junit4]   2> 1548209 T7629 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1548209 T7629 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/
   [junit4]   2> 1548209 T7629 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 1548210 T7629 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data
   [junit4]   2> 1548210 T7629 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/index/
   [junit4]   2> 1548210 T7629 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1548211 T7629 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/index
   [junit4]   2> 1548211 T7629 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=45.2412109375, floorSegmentMB=2.087890625, forceMergeDeletesPctAllowed=3.6099600856064686, segmentsPerTier=34.0, maxCFSSegmentSizeMB=1.51953125, noCFSRatio=0.3231567203580047
   [junit4]   2> 1548212 T7629 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5be5ac0b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4382354f),segFN=segments_1,generation=1}
   [junit4]   2> 1548212 T7629 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1548213 T7629 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1548214 T7629 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1548214 T7629 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1548214 T7629 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1548215 T7629 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1548215 T7629 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1548215 T7629 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1548216 T7629 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1548216 T7629 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1548217 T7629 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1548217 T7629 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1548217 T7629 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1548218 T7629 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1548218 T7629 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1548218 T7629 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 1548219 T7629 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1548219 T7629 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1548248 T7629 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1548248 T7629 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1548248 T7629 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1548249 T7629 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1548249 T7629 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1548268 T7629 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548270 T7629 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548272 T7629 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548274 T7629 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548276 T7629 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1548277 T7629 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1548277 T7629 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=49.8701171875, floorSegmentMB=0.328125, forceMergeDeletesPctAllowed=3.234204313587825, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1548278 T7629 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5be5ac0b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4382354f),segFN=segments_1,generation=1}
   [junit4]   2> 1548278 T7629 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1548278 T7629 oass.SolrIndexSearcher.<init> Opening Searcher@24ab2e62[collection1] main
   [junit4]   2> 1548279 T7629 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/conf
   [junit4]   2> 1548279 T7629 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/conf}
   [junit4]   2> 1548280 T7629 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/conf
   [junit4]   2> 1548281 T7629 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 1548281 T7629 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/conf
   [junit4]   2> 1548281 T7629 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1548282 T7629 oash.ReplicationHandler.inform Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1548282 T7629 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1548282 T7629 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1548282 T7630 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24ab2e62[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1548283 T7619 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1548284 T7619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1548300 T7619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1548305 T7619 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:43584
   [junit4]   2> 1548306 T7619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1548306 T7619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1548306 T7619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002
   [junit4]   2> 1548307 T7619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/'
   [junit4]   2> 1548352 T7619 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/solr.xml
   [junit4]   2> 1548434 T7619 oasc.CoreContainer.<init> New CoreContainer 1527518381
   [junit4]   2> 1548435 T7619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/]
   [junit4]   2> 1548436 T7619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1548437 T7619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1548437 T7619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1548437 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1548438 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1548438 T7619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1548438 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1548439 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1548439 T7619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1548440 T7619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1548441 T7619 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1548441 T7619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1548441 T7619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1548442 T7619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1548446 T7641 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/'
   [junit4]   2> 1548499 T7641 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1548542 T7641 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1548543 T7641 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/conf/schema.xml
   [junit4]   2> 1548550 T7641 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1548567 T7641 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1548568 T7641 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/
   [junit4]   2> 1548568 T7641 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1548568 T7641 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/
   [junit4]   2> 1548569 T7641 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 1548569 T7641 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data
   [junit4]   2> 1548569 T7641 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index/
   [junit4]   2> 1548570 T7641 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1548571 T7641 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index
   [junit4]   2> 1548572 T7641 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=45.2412109375, floorSegmentMB=2.087890625, forceMergeDeletesPctAllowed=3.6099600856064686, segmentsPerTier=34.0, maxCFSSegmentSizeMB=1.51953125, noCFSRatio=0.3231567203580047
   [junit4]   2> 1548573 T7641 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@750436e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@67b0f55c),segFN=segments_1,generation=1}
   [junit4]   2> 1548573 T7641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1548574 T7641 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1548575 T7641 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1548575 T7641 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1548575 T7641 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1548576 T7641 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1548576 T7641 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1548576 T7641 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1548577 T7641 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1548577 T7641 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1548578 T7641 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1548578 T7641 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1548578 T7641 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1548579 T7641 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1548579 T7641 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1548579 T7641 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1548580 T7641 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 1548580 T7641 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1548580 T7641 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1548581 T7641 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1548581 T7641 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1548581 T7641 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1548581 T7641 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1548583 T7641 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548585 T7641 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548587 T7641 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548590 T7641 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548592 T7641 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1548592 T7641 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1548593 T7641 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=49.8701171875, floorSegmentMB=0.328125, forceMergeDeletesPctAllowed=3.234204313587825, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1548593 T7641 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@750436e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@67b0f55c),segFN=segments_1,generation=1}
   [junit4]   2> 1548594 T7641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1548594 T7641 oass.SolrIndexSearcher.<init> Opening Searcher@18f95768[collection1] main
   [junit4]   2> 1548595 T7641 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/conf
   [junit4]   2> 1548595 T7641 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/conf}
   [junit4]   2> 1548595 T7641 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/conf
   [junit4]   2> 1548596 T7641 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 1548597 T7641 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/conf
   [junit4]   2> 1548597 T7641 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1548598 T7641 oash.SnapPuller.startExecutorService Poll Scheduled at an interval of 1000ms
   [junit4]   2> 1548598 T7642 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18f95768[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1548598 T7641 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1548599 T7619 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1548599 T7619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C2416 name=collection1 org.apache.solr.core.SolrCore@2516d7e
   [junit4]   2> 1548603 T7622 C2416 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&sort=id+desc&version=2} hits=0 status=0 QTime=1 
   [junit4]   2> 1548606 T7622 C2416 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=details&version=2&qt=/replication&_trace=getDetails} status=0 QTime=0 
   [junit4]   2> 1548611 T7623 C2416 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&slave=false&command=details&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> ASYNC  NEW_CORE C2417 name=collection1 org.apache.solr.core.SolrCore@3368dd78
   [junit4]   2> 1548612 T7634 C2417 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=details&version=2&qt=/replication&_trace=getDetails} status=0 QTime=4 
   [junit4]   2> 1548616 T7622 C2416 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[0]} 0 1
   [junit4]   2> 1548618 T7622 C2416 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1548621 T7622 C2416 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5be5ac0b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4382354f),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5be5ac0b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4382354f),segFN=segments_2,generation=2}
   [junit4]   2> 1548622 T7622 C2416 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 1548624 T7622 C2416 oass.SolrIndexSearcher.<init> Opening Searcher@6f39ae5d[collection1] main
   [junit4]   2> 1548625 T7622 C2416 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1548625 T7630 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6f39ae5d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.0.0):c1)))}
   [junit4]   2> 1548626 T7622 C2416 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&softCommit=false&version=2&waitSearcher=true&commit=true} {commit=} 0 8
   [junit4]   2> 1548685 T7645 oash.SnapPuller.<init> WARN 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1548687 T7645 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 1548689 T7623 C2416 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=indexversion&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1548690 T7645 oash.SnapPuller.fetchLatestIndex Master's generation: 2
   [junit4]   2> 1548690 T7645 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
   [junit4]   2> 1548690 T7645 oash.SnapPuller.fetchLatestIndex Starting replication process
   [junit4]   2> ASYNC  NEW_CORE C2418 name=collection1 org.apache.solr.core.SolrCore@2516d7e
   [junit4]   2> 1548714 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=filelist&version=2&qt=/replication&generation=2} status=0 QTime=21 
   [junit4]   2> 1548715 T7645 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 4
   [junit4]   2> 1548716 T7645 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721687
   [junit4]   2> 1548716 T7645 oasu.DefaultSolrCoreState.closeIndexWriter Closing IndexWriter...
   [junit4]   2> 1548716 T7645 oasu.DefaultSolrCoreState.closeIndexWriter Waiting until IndexWriter is unused... core=collection1
   [junit4]   2> 1548716 T7645 oasu.DefaultSolrCoreState.closeIndexWriter Rollback old IndexWriter... core=collection1
   [junit4]   2> 1548717 T7645 oash.SnapPuller.fetchLatestIndex Starting download to MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@3a0c82a6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@b557e10; maxCacheMB=0.6826171875 maxMergeSizeMB=0.3369140625)) fullCopy=false
   [junit4]   2> 1548720 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=2&file=segments_2} status=0 QTime=0 
   [junit4]   2> 1548724 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=2&file=_0.cfe} status=0 QTime=0 
   [junit4]   2> 1548727 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=2&file=_0.si} status=0 QTime=0 
   [junit4]   2> 1548730 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=2&file=_0.cfs} status=0 QTime=0 
   [junit4]   2> 1548731 T7645 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
   [junit4]   2> 1548737 T7645 oasu.DefaultSolrCoreState.openIndexWriter Creating new IndexWriter...
   [junit4]   2> 1548738 T7645 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1548739 T7645 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@750436e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@67b0f55c),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@750436e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@67b0f55c),segFN=segments_2,generation=2}
   [junit4]   2> 1548741 T7645 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 1548741 T7645 oasu.DefaultSolrCoreState.openIndexWriter New IndexWriter is ready to be used.
   [junit4]   2> 1548743 T7645 oass.SolrIndexSearcher.<init> Opening Searcher@1ee7e5e0[collection1] main
   [junit4]   2> 1548743 T7642 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ee7e5e0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.0.0):c1)))}
   [junit4]   2> 1548744 T7645 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721687 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721687;done=true>>]
   [junit4]   2> 1548744 T7645 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721687
   [junit4]   2> ASYNC  NEW_CORE C2419 name=collection1 org.apache.solr.core.SolrCore@3368dd78
   [junit4]   2> 1548744 T7635 C2419 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=fetchindex&wait=true&masterUrl=http://127.0.0.1:43367/solr/replication} status=0 QTime=60 
   [junit4]   2> 1548755 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&slave=false&command=details&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1548756 T7634 C2419 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=details&version=2&qt=/replication&_trace=getDetails} status=0 QTime=4 
   [junit4]   2> 1548760 T7622 C2418 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 1
   [junit4]   2> 1548762 T7622 C2418 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1548765 T7622 C2418 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5be5ac0b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4382354f),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5be5ac0b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4382354f),segFN=segments_3,generation=3}
   [junit4]   2> 1548766 T7622 C2418 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 1548767 T7622 C2418 oass.SolrIndexSearcher.<init> Opening Searcher@23dcdb17[collection1] main
   [junit4]   2> 1548768 T7622 C2418 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1548768 T7630 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23dcdb17[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.0.0):c1) Uninverting(_1(5.0.0):c1)))}
   [junit4]   2> 1548768 T7622 C2418 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&softCommit=false&version=2&waitSearcher=true&commit=true} {commit=} 0 6
   [junit4]   2> 1548771 T7648 oash.SnapPuller.<init> WARN 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1548772 T7648 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 1548774 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=indexversion&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1548775 T7648 oash.SnapPuller.fetchLatestIndex Master's generation: 3
   [junit4]   2> 1548775 T7648 oash.SnapPuller.fetchLatestIndex Slave's generation: 2
   [junit4]   2> 1548775 T7648 oash.SnapPuller.fetchLatestIndex Starting replication process
   [junit4]   2> 1548777 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=filelist&version=2&qt=/replication&generation=3} status=0 QTime=0 
   [junit4]   2> 1548778 T7648 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 7
   [junit4]   2> 1548779 T7648 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721750
   [junit4]   2> 1548780 T7648 oasu.DefaultSolrCoreState.closeIndexWriter Closing IndexWriter...
   [junit4]   2> 1548780 T7648 oasu.DefaultSolrCoreState.closeIndexWriter Waiting until IndexWriter is unused... core=collection1
   [junit4]   2> 1548781 T7648 oasu.DefaultSolrCoreState.closeIndexWriter Rollback old IndexWriter... core=collection1
   [junit4]   2> 1548782 T7648 oash.SnapPuller.fetchLatestIndex Starting download to MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@35f29e3d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4b72d276; maxCacheMB=0.6826171875 maxMergeSizeMB=0.3369140625)) fullCopy=false
   [junit4]   2> 1548785 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=3&file=_1.si} status=0 QTime=0 
   [junit4]   2> 1548789 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=3&file=segments_3} status=0 QTime=0 
   [junit4]   2> 1548790 T7648 oash.SnapPuller.downloadIndexFiles Skipping download for _0.cfe because it already exists
   [junit4]   2> 1548792 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=3&file=_1.cfe} status=0 QTime=0 
   [junit4]   2> 1548793 T7648 oash.SnapPuller.downloadIndexFiles Skipping download for _0.si because it already exists
   [junit4]   2> 1548795 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=3&file=_1.cfs} status=0 QTime=0 
   [junit4]   2> 1548796 T7648 oash.SnapPuller.downloadIndexFiles Skipping download for _0.cfs because it already exists
   [junit4]   2> 1548796 T7648 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
   [junit4]   2> 1548798 T7648 oasu.DefaultSolrCoreState.openIndexWriter Creating new IndexWriter...
   [junit4]   2> 1548799 T7648 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1548800 T7648 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@750436e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@67b0f55c),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@750436e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@67b0f55c),segFN=segments_3,generation=3}
   [junit4]   2> 1548801 T7648 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
   [junit4]   2> 1548801 T7648 oasu.DefaultSolrCoreState.openIndexWriter New IndexWriter is ready to be used.
   [junit4]   2> 1548803 T7648 oass.SolrIndexSearcher.<init> Opening Searcher@6b7076ec[collection1] main
   [junit4]   2> 1548803 T7642 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b7076ec[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.0.0):c1) Uninverting(_1(5.0.0):c1)))}
   [junit4]   2> 1548804 T7648 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721750 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721750;done=true>>]
   [junit4]   2> 1548804 T7648 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721750
   [junit4]   2> 1548804 T7635 C2419 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=fetchindex&wait=true&masterUrl=http://127.0.0.1:43367/solr/replication} status=0 QTime=33 
   [junit4]   2> 1548810 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&slave=false&command=details&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1548811 T7634 C2419 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=details&version=2&qt=/replication&_trace=getDetails} status=0 QTime=4 
   [junit4]   2> 1548815 T7622 C2418 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 1
   [junit4]   2> 1548817 T7622 C2418 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1548821 T7622 C2418 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5be5ac0b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4382354f),segFN=segments_2,generation=2}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5be5ac0b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4382354f),segFN=segments_3,generation=3}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@5be5ac0b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4382354f),segFN=segments_4,generation=4}
   [junit4]   2> 1548822 T7622 C2418 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
   [junit4]   2> 1548823 T7622 C2418 oass.SolrIndexSearcher.<init> Opening Searcher@59ebdb1c[collection1] main
   [junit4]   2> 1548823 T7622 C2418 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 1548824 T7630 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@59ebdb1c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.0.0):c1) Uninverting(_1(5.0.0):c1) Uninverting(_2(5.0.0):c1)))}
   [junit4]   2> 1548824 T7622 C2418 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={wt=javabin&softCommit=false&version=2&waitSearcher=true&commit=true} {commit=} 0 7
   [junit4]   2> 1548828 T7650 oash.SnapPuller.<init> WARN 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1548829 T7650 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 1548831 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=indexversion&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1548831 T7650 oash.SnapPuller.fetchLatestIndex Master's generation: 4
   [junit4]   2> 1548832 T7650 oash.SnapPuller.fetchLatestIndex Slave's generation: 3
   [junit4]   2> 1548832 T7650 oash.SnapPuller.fetchLatestIndex Starting replication process
   [junit4]   2> 1548834 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=filelist&version=2&qt=/replication&generation=4} status=0 QTime=0 
   [junit4]   2> 1548835 T7650 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 10
   [junit4]   2> 1548836 T7650 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721807
   [junit4]   2> 1548836 T7650 oasu.DefaultSolrCoreState.closeIndexWriter Closing IndexWriter...
   [junit4]   2> 1548836 T7650 oasu.DefaultSolrCoreState.closeIndexWriter Waiting until IndexWriter is unused... core=collection1
   [junit4]   2> 1548836 T7650 oasu.DefaultSolrCoreState.closeIndexWriter Rollback old IndexWriter... core=collection1
   [junit4]   2> 1548838 T7650 oash.SnapPuller.fetchLatestIndex Starting download to MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@9943b0a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@62b0126f; maxCacheMB=0.6826171875 maxMergeSizeMB=0.3369140625)) fullCopy=false
   [junit4]   2> 1548838 T7650 oash.SnapPuller.downloadIndexFiles Skipping download for _1.si because it already exists
   [junit4]   2> 1548841 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=4&file=segments_4} status=0 QTime=0 
   [junit4]   2> 1548842 T7650 oash.SnapPuller.downloadIndexFiles Skipping download for _0.cfe because it already exists
   [junit4]   2> 1548842 T7650 oash.SnapPuller.downloadIndexFiles Skipping download for _1.cfe because it already exists
   [junit4]   2> 1548845 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=4&file=_2.si} status=0 QTime=0 
   [junit4]   2> 1548846 T7650 oash.SnapPuller.downloadIndexFiles Skipping download for _0.si because it already exists
   [junit4]   2> 1548846 T7650 oash.SnapPuller.downloadIndexFiles Skipping download for _1.cfs because it already exists
   [junit4]   2> 1548849 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=4&file=_2.cfs} status=0 QTime=0 
   [junit4]   2> 1548850 T7650 oash.SnapPuller.downloadIndexFiles Skipping download for _0.cfs because it already exists
   [junit4]   2> 1548852 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=filestream&checksum=true&command=filecontent&qt=/replication&generation=4&file=_2.cfe} status=0 QTime=0 
   [junit4]   2> 1548853 T7650 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
   [junit4]   2> 1548855 T7650 oasu.DefaultSolrCoreState.openIndexWriter Creating new IndexWriter...
   [junit4]   2> 1548856 T7650 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1548908 T7650 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@750436e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@67b0f55c),segFN=segments_3,generation=3}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@750436e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@67b0f55c),segFN=segments_4,generation=4}
   [junit4]   2> 1548909 T7650 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
   [junit4]   2> 1548909 T7650 oasu.DefaultSolrCoreState.openIndexWriter New IndexWriter is ready to be used.
   [junit4]   2> 1548912 T7650 oass.SolrIndexSearcher.<init> Opening Searcher@7263f98d[collection1] main
   [junit4]   2> 1548912 T7642 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7263f98d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(5.0.0):c1) Uninverting(_1(5.0.0):c1) Uninverting(_2(5.0.0):c1)))}
   [junit4]   2> 1548912 T7650 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721807 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721807;done=true>>]
   [junit4]   2> 1548913 T7650 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index.20141206045721807
   [junit4]   2> 1548913 T7636 C2419 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={command=fetchindex&wait=true&masterUrl=http://127.0.0.1:43367/solr/replication} status=0 QTime=85 
   [junit4]   2> 1548930 T7619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1548934 T7619 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:61059
   [junit4]   2> 1548935 T7619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1548935 T7619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1548936 T7619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003
   [junit4]   2> 1548936 T7619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/'
   [junit4]   2> 1548984 T7619 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/solr.xml
   [junit4]   2> 1549032 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=indexversion&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1549033 T7643 oash.SnapPuller.fetchLatestIndex Slave in sync with master.
   [junit4]   2> 1549079 T7619 oasc.CoreContainer.<init> New CoreContainer 1022407322
   [junit4]   2> 1549080 T7619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/]
   [junit4]   2> 1549081 T7619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1549081 T7619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1549082 T7619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1549082 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1549082 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1549083 T7619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1549083 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1549083 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1549084 T7619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1549084 T7619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1549085 T7619 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1549085 T7619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1549086 T7619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1549086 T7619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1549091 T7661 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/'
   [junit4]   2> 1549152 T7661 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1549201 T7661 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1549202 T7661 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/conf/schema.xml
   [junit4]   2> 1549210 T7661 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1549229 T7661 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1549230 T7661 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/
   [junit4]   2> 1549230 T7661 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1549231 T7661 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data/
   [junit4]   2> 1549231 T7661 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 1549231 T7661 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data
   [junit4]   2> 1549232 T7661 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data/index/
   [junit4]   2> 1549232 T7661 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1549232 T7661 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data/index
   [junit4]   2> 1549233 T7661 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=45.2412109375, floorSegmentMB=2.087890625, forceMergeDeletesPctAllowed=3.6099600856064686, segmentsPerTier=34.0, maxCFSSegmentSizeMB=1.51953125, noCFSRatio=0.3231567203580047
   [junit4]   2> 1549234 T7661 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a610184 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66aa866e),segFN=segments_1,generation=1}
   [junit4]   2> 1549234 T7661 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1549235 T7661 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1549235 T7661 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1549236 T7661 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1549236 T7661 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1549236 T7661 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1549237 T7661 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1549237 T7661 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1549237 T7661 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1549238 T7661 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1549238 T7661 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1549239 T7661 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1549239 T7661 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1549239 T7661 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1549240 T7661 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1549240 T7661 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1549240 T7661 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 1549241 T7661 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1549241 T7661 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1549241 T7661 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1549241 T7661 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1549242 T7661 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1549242 T7661 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1549261 T7661 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1549264 T7661 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1549266 T7661 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1549269 T7661 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1549272 T7661 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1549272 T7661 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1549272 T7661 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=49.8701171875, floorSegmentMB=0.328125, forceMergeDeletesPctAllowed=3.234204313587825, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1549273 T7661 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@a610184 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66aa866e),segFN=segments_1,generation=1}
   [junit4]   2> 1549273 T7661 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1549274 T7661 oass.SolrIndexSearcher.<init> Opening Searcher@56bdb812[collection1] main
   [junit4]   2> 1549274 T7661 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/conf
   [junit4]   2> 1549274 T7661 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/conf}
   [junit4]   2> 1549275 T7661 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/conf
   [junit4]   2> 1549276 T7661 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 1549277 T7661 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/conf
   [junit4]   2> 1549277 T7661 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1549277 T7661 oash.SnapPuller.<init> WARN 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1549277 T7661 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
   [junit4]   2> 1549278 T7661 oash.ReplicationHandler.inform Replication enabled for following config files: schema.xml
   [junit4]   2> 1549278 T7661 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1549278 T7661 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1549278 T7662 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@56bdb812[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1549279 T7619 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1549279 T7619 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1549285 T7624 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&slave=false&command=details&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> ASYNC  NEW_CORE C2420 name=collection1 org.apache.solr.core.SolrCore@3de85990
   [junit4]   2> 1549286 T7654 C2420 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=details&version=2&qt=/replication&_trace=getDetails} status=0 QTime=4 
   [junit4]   2> 1549287 T7619 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1022407322
   [junit4]   2> 1549290 T7619 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3de85990
   [junit4]   2> 1549290 T7619 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 1549291 T7619 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1549291 T7619 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1549291 T7619 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1549292 T7619 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1549293 T7619 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1549293 T7619 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data/index;done=false>>]
   [junit4]   2> 1549293 T7619 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data/index
   [junit4]   2> 1549294 T7619 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data;done=false>>]
   [junit4]   2> 1549294 T7619 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-003/collection1/data
   [junit4]   2> 1549295 T7619 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 1550032 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=indexversion&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1550033 T7643 oash.SnapPuller.fetchLatestIndex Slave in sync with master.
   [junit4]   2> 1551031 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=indexversion&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1551032 T7643 oash.SnapPuller.fetchLatestIndex Slave in sync with master.
   [junit4]   2> 1552031 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=indexversion&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1552032 T7643 oash.SnapPuller.fetchLatestIndex Slave in sync with master.
   [junit4]   2> 1553031 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=indexversion&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1553032 T7643 oash.SnapPuller.fetchLatestIndex Slave in sync with master.
   [junit4]   2> 1554031 T7623 C2418 oasc.SolrCore.execute [collection1] webapp=/solr path=/replication params={wt=javabin&command=indexversion&version=2&qt=/replication} status=0 QTime=0 
   [junit4]   2> 1554032 T7643 oash.SnapPuller.fetchLatestIndex Slave in sync with master.
   [junit4]   2> 1554293 T7619 oas.SolrTestCaseJ4.tearDown ###Ending doTestDetails
   [junit4]   2> 1554293 T7619 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=231193030
   [junit4]   2> 1554297 T7619 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2516d7e
   [junit4]   2> 1554297 T7619 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=3,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 1554299 T7619 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1554300 T7619 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1554300 T7619 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1554302 T7619 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1554303 T7619 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1554304 T7619 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data;done=false>>]
   [junit4]   2> 1554304 T7619 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data
   [junit4]   2> 1554305 T7619 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/index;done=false>>]
   [junit4]   2> 1554305 T7619 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/index
   [junit4]   2> 1554306 T7619 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 1559298 T7619 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1527518381
   [junit4]   2> 1559301 T7619 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3368dd78
   [junit4]   2> 1570030 T7643 oash.SnapPuller.fetchLatestIndex ERROR Master at: http://127.0.0.1:43367/solr is not available. Index fetch failed. Exception: IOException occured when talking to server at: http://127.0.0.1:43367/solr
   [junit4]   2> 1570033 T7619 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 1570033 T7619 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1570034 T7619 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1570034 T7619 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1570036 T7619 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1570037 T7619 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1570037 T7619 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index;done=false>>]
   [junit4]   2> 1570038 T7619 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data/index
   [junit4]   2> 1570038 T7619 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data;done=false>>]
   [junit4]   2> 1570039 T7619 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-002/collection1/data
   [junit4]   2> 1570040 T7619 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 1574321 T7619 oas.SolrTestCaseJ4.setUp ###Starting doTestIndexAndConfigReplication
   [junit4]   2> 1574326 T7619 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1574329 T7619 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:36115
   [junit4]   2> 1574329 T7619 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1574330 T7619 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1574330 T7619 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001
   [junit4]   2> 1574330 T7619 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/'
   [junit4]   2> 1574344 T7619 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/solr.xml
   [junit4]   2> 1574416 T7619 oasc.CoreContainer.<init> New CoreContainer 892647751
   [junit4]   2> 1574416 T7619 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/]
   [junit4]   2> 1574418 T7619 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1574418 T7619 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1574418 T7619 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1574419 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1574419 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1574419 T7619 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1574420 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1574420 T7619 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1574420 T7619 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1574421 T7619 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1574421 T7619 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1574422 T7619 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1574422 T7619 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1574422 T7619 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1574428 T7673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/'
   [junit4]   2> 1574467 T7673 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1574501 T7673 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1574501 T7673 oass.IndexSchema.readSchema Reading Solr Schema from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/conf/schema.xml
   [junit4]   2> 1574506 T7673 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1574517 T7673 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1574518 T7673 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/
   [junit4]   2> 1574518 T7673 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1574518 T7673 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/
   [junit4]   2> 1574518 T7673 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 1574520 T7673 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data
   [junit4]   2> 1574520 T7673 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/index/
   [junit4]   2> 1574520 T7673 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1574520 T7673 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/data/index
   [junit4]   2> 1574521 T7673 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=45.2412109375, floorSegmentMB=2.087890625, forceMergeDeletesPctAllowed=3.6099600856064686, segmentsPerTier=34.0, maxCFSSegmentSizeMB=1.51953125, noCFSRatio=0.3231567203580047
   [junit4]   2> 1574521 T7673 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@40f05360 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@61f1e4ee),segFN=segments_1,generation=1}
   [junit4]   2> 1574522 T7673 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1574522 T7673 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1574523 T7673 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1574523 T7673 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1574523 T7673 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1574523 T7673 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1574523 T7673 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1574524 T7673 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1574524 T7673 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1574524 T7673 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1574525 T7673 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1574525 T7673 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1574525 T7673 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1574525 T7673 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1574526 T7673 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1574526 T7673 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 1574526 T7673 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1574526 T7673 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1574526 T7673 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1574527 T7673 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 1574527 T7673 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 1574527 T7673 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 1574527 T7673 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1574529 T7673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1574530 T7673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1574532 T7673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1574533 T7673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1574535 T7673 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1574535 T7673 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1574536 T7673 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=49.8701171875, floorSegmentMB=0.328125, forceMergeDeletesPctAllowed=3.234204313587825, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1574536 T7673 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@40f05360 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@61f1e4ee),segFN=segments_1,generation=1}
   [junit4]   2> 1574536 T7673 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1574537 T7673 oass.SolrIndexSearcher.<init> Opening Searcher@7359d730[collection1] main
   [junit4]   2> 1574537 T7673 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/collection1/conf
   [junit4]   2> 1574537 T7673 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.handler.TestReplicationHandler-4711AD7639F6F7-001/solr-instance-001/

[...truncated too long message...]

lrServer.java:116)
   [junit4]   2> 	        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
   [junit4]   2> 	        at org.apache.solr.handler.TestReplicationHandler.index(TestReplicationHandler.java:173)
   [junit4]   2> 	        at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1144)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	        at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
   [junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	   6) Thread[id=8488, name=Thread-4108, state=WAITING, group=TGRP-TestReplicationHandler]
   [junit4]   2> 	        at java.lang.Object.wait(Native Method)
   [junit4]   2> 	        at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	        at org.apache.solr.core.CloserThread.run(CoreContainer.java:924)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {name=PostingsFormat(name=LuceneVarGapDocFreqInterval), newname=PostingsFormat(name=LuceneVarGapDocFreqInterval), id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{}, sim=DefaultSimilarity, locale=ja_JP_JP_#u-ca-japanese, timezone=Etc/GMT-9
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=7,free=306206864,total=525336576
   [junit4]   2> NOTE: All tests run in this JVM: [EchoParamsTest, SuggesterTest, TestSchemaSimilarityResource, CollectionsAPIDistributedZkTest, TestCollapseQParserPlugin, SuggesterTSTTest, TestSolrConfigHandlerCloud, TestCloudPivotFacet, TestMergePolicyConfig, TestSolrConfigHandlerConcurrent, DOMUtilTest, OutputWriterTest, VMParamsZkACLAndCredentialsProvidersTest, ClusterStateUpdateTest, TestManagedResourceStorage, SolrInfoMBeanTest, TestConfigReload, AsyncMigrateRouteKeyTest, PluginInfoTest, SpellPossibilityIteratorTest, FieldMutatingUpdateProcessorTest, DistribCursorPagingTest, TestHighFrequencyDictionaryFactory, TestFreeTextSuggestions, TestLRUCache, TestInitParams, DocumentAnalysisRequestHandlerTest, CoreAdminHandlerTest, TestSolrXmlPersistence, IndexBasedSpellCheckerTest, TestDistributedGrouping, StandardRequestHandlerTest, MergeStrategyTest, TestSchemaManager, HdfsBasicDistributedZkTest, BlockDirectoryTest, BadIndexSchemaTest, TimeZoneUtilsTest, ResponseHeaderTest, DistributedDebugComponentTest, OutOfBoxZkACLAndCredentialsProvidersTest, ResponseLogComponentTest, TermVectorComponentDistributedTest, EnumFieldTest, DistributedTermsComponentTest, TestCloudSchemaless, WordBreakSolrSpellCheckerTest, CoreContainerCoreInitFailuresTest, TestCSVLoader, FileBasedSpellCheckerTest, TestDocSet, LeaderInitiatedRecoveryOnCommitTest, SolrIndexSplitterTest, TestStandardQParsers, LeaderElectionTest, ShardRoutingTest, TestReplicationHandler]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestReplicationHandler -Dtests.seed=4711AD7639F6F7 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ja_JP_JP_#u-ca-japanese -Dtests.timezone=Etc/GMT-9 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J3 | TestReplicationHandler (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4711AD7639F6F7]:0)
   [junit4] Completed on J3 in 7223.39s, 13 tests, 2 errors <<< FAILURES!

[...truncated 13 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:536: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:477: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:190: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:508: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1363: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:970: There were test failures: 450 suites, 1843 tests, 1 suite-level error, 1 error, 56 ignored (10 assumptions)

Total time: 234 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #689
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 837177060 bytes
Compression is 0.0%
Took 2 min 24 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 693 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/693/

No tests ran.

Build Log:
[...truncated 4 lines...]
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
	... 35 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
ERROR: Subversion update failed
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x at hudson.remoting.Channel@58e1bc28:lucene
	at hudson.FilePath.act(FilePath.java:916)
	at hudson.FilePath.act(FilePath.java:893)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
	at hudson.model.Run.execute(Run.java:1706)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:211)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
	... 14 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	... 13 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
	... 35 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
Retrying after 10 seconds
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/.
Updating http://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x at revision '2014-12-04T14:04:03.652 +0000'
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
	... 35 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
ERROR: Subversion update failed
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x at hudson.remoting.Channel@58e1bc28:lucene
	at hudson.FilePath.act(FilePath.java:916)
	at hudson.FilePath.act(FilePath.java:893)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
	at hudson.model.Run.execute(Run.java:1706)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:211)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
	... 14 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	... 13 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
	... 35 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
Retrying after 10 seconds
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/.
Updating http://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x at revision '2014-12-04T14:04:03.652 +0000'
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
	... 35 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
ERROR: Subversion update failed
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x at hudson.remoting.Channel@58e1bc28:lucene
	at hudson.FilePath.act(FilePath.java:916)
	at hudson.FilePath.act(FilePath.java:893)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
	at hudson.model.Run.execute(Run.java:1706)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:211)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
	... 14 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	... 13 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: connection refused by the server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:520)
	... 35 more
Caused by: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #689
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 464 bytes
Compression is 0.0%
Took 0.19 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 692 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/692/

No tests ran.

Build Log:
[...truncated 112 lines...]
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:514)
	... 35 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
ERROR: Subversion update failed
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x at hudson.remoting.Channel@56a2258b:lucene
	at hudson.FilePath.act(FilePath.java:916)
	at hudson.FilePath.act(FilePath.java:893)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
	at hudson.model.Run.execute(Run.java:1706)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:211)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
	... 14 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	... 13 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:514)
	... 35 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
Retrying after 10 seconds
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/.
Updating http://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x at revision '2014-12-03T16:22:40.776 +0000'
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:514)
	... 35 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
ERROR: Subversion update failed
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x at hudson.remoting.Channel@56a2258b:lucene
	at hudson.FilePath.act(FilePath.java:916)
	at hudson.FilePath.act(FilePath.java:893)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
	at hudson.model.Run.execute(Run.java:1706)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:211)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
	... 14 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	... 13 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:514)
	... 35 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
Retrying after 10 seconds
Cleaning up /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/.
Updating http://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x at revision '2014-12-03T16:22:40.776 +0000'
ERROR: Failed to update http://svn.apache.org/repos/asf/lucene/dev/branches/branch_5x
org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:514)
	... 35 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
ERROR: Subversion update failed
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x at hudson.remoting.Channel@56a2258b:lucene
	at hudson.FilePath.act(FilePath.java:916)
	at hudson.FilePath.act(FilePath.java:893)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:936)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:871)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1252)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524)
	at hudson.model.Run.execute(Run.java:1706)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:232)
Caused by: java.io.IOException
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:211)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:161)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:1030)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:1011)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:987)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2474)
	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:328)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: hudson.scm.subversion.UpdaterException: failed to perform svn update
	... 14 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:388)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:361)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:707)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:627)
	at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:102)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:1020)
	at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getRepositoryUUID(DAVRepository.java:148)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:339)
	at org.tmatesoft.svn.core.internal.wc16.SVNBasicDelegate.createRepository(SVNBasicDelegate.java:328)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.update(SVNUpdateClient16.java:482)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:364)
	at org.tmatesoft.svn.core.internal.wc16.SVNUpdateClient16.doUpdate(SVNUpdateClient16.java:274)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:27)
	at org.tmatesoft.svn.core.internal.wc2.old.SvnOldUpdate.run(SvnOldUpdate.java:11)
	at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20)
	at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1238)
	at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:294)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:311)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:291)
	at org.tmatesoft.svn.core.wc.SVNUpdateClient.doUpdate(SVNUpdateClient.java:387)
	at hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:157)
	... 13 more
Caused by: svn: E175002: OPTIONS /repos/asf/lucene/dev/branches/branch_5x failed
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154)
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97)
	... 35 more
Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64)
	at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:777)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:382)
	... 34 more
Caused by: svn: E175002: OPTIONS request failed on '/repos/asf/lucene/dev/branches/branch_5x'
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:775)
	... 35 more
Caused by: svn: E175002: timed out waiting for server
	at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:514)
	... 35 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:579)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.connect(SVNSocketFactory.java:146)
	at org.tmatesoft.svn.core.internal.util.SVNSocketFactory.createPlainSocket(SVNSocketFactory.java:73)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.connect(HTTPConnection.java:280)
	at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:451)
	... 35 more
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #689
Archived 1 artifacts
Archive block size is 32768
Received 0 blocks and 464 bytes
Compression is 0.0%
Took 0.16 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-5.x - Build # 691 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-5.x/691/

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
reload the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: reload the collection time out:180s
	at __randomizedtesting.SeedInfo.seed([19B2B8D85CD3422A:985436C02B8C2216]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:569)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.request.CollectionAdminRequest.process(CollectionAdminRequest.java:379)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testSolrJAPICalls(CollectionsAPIDistributedZkTest.java:332)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:201)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10171 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/init-core-data-001
   [junit4]   2> 1115102 T3661 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1115103 T3661 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1115111 T3661 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1115112 T3661 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1115113 T3662 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1115213 T3661 oasc.ZkTestServer.run start zk server on port:59089
   [junit4]   2> 1115214 T3661 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1115216 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1115220 T3669 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fbeab97 name:ZooKeeperConnection Watcher:127.0.0.1:59089 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1115220 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1115221 T3661 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1115221 T3661 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1115233 T3661 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1115234 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1115236 T3672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6aea681 name:ZooKeeperConnection Watcher:127.0.0.1:59089/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1115236 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1115236 T3661 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1115237 T3661 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1115239 T3661 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1115242 T3661 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1115244 T3661 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1115247 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1115248 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1115253 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1115254 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1115358 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1115358 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1115361 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1115362 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1115365 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1115366 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1115368 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1115369 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1115372 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1115373 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1115376 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1115377 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1115380 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1115381 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1115393 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1115395 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1115398 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1115399 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 1115403 T3661 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1115404 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1115406 T3675 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d357e51 name:ZooKeeperConnection Watcher:127.0.0.1:59089 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1115406 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1115406 T3661 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1115407 T3661 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1115409 T3661 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1115410 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1115412 T3678 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@80f8d8a name:ZooKeeperConnection Watcher:127.0.0.1:59089/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1115412 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1115413 T3661 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1115415 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 1115416 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 1115419 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 1115420 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 1115524 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1115525 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1115527 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 1115528 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 1115531 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 1115531 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 1115534 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 1115534 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 1115537 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1115538 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1115540 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1115541 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1115549 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 1115551 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 1115553 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 1115553 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 1115556 T3661 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 1115557 T3661 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 1117479 T3661 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1117482 T3661 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:61218
   [junit4]   2> 1117482 T3661 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1117483 T3661 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1117484 T3661 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-002
   [junit4]   2> 1117484 T3661 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-002/'
   [junit4]   2> 1117554 T3661 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-002/solr.xml
   [junit4]   2> 1117666 T3661 oasc.CoreContainer.<init> New CoreContainer 1736521617
   [junit4]   2> 1117666 T3661 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-002/]
   [junit4]   2> 1117685 T3661 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1117685 T3661 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1117686 T3661 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1117686 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1117686 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1117687 T3661 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1117687 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1117687 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1117688 T3661 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1117688 T3661 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1117689 T3661 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1117689 T3661 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1117690 T3661 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1117690 T3661 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1117690 T3661 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59089/solr
   [junit4]   2> 1117691 T3661 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1117691 T3661 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1117692 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1117694 T3690 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@662a6893 name:ZooKeeperConnection Watcher:127.0.0.1:59089 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1117695 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1117696 T3661 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1117698 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1117700 T3693 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1edc33f8 name:ZooKeeperConnection Watcher:127.0.0.1:59089/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1117701 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1117703 T3661 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1117706 T3661 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1117710 T3661 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1117714 T3661 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1117717 T3661 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1117721 T3661 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1117724 T3661 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61218_
   [junit4]   2> 1117725 T3661 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:61218_
   [junit4]   2> 1117728 T3661 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1117731 T3661 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1117734 T3661 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1117737 T3661 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:61218_
   [junit4]   2> 1117737 T3661 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1117740 T3661 oasc.Overseer.start Overseer (id=92900272846012421-127.0.0.1:61218_-n_0000000000) starting
   [junit4]   2> 1117743 T3661 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1117753 T3661 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1117754 T3695 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1117755 T3661 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1117758 T3661 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1117760 T3661 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1117764 T3694 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1117770 T3661 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1117770 T3661 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1117771 T3661 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1117772 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1117782 T3700 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f4bc04a name:ZooKeeperConnection Watcher:127.0.0.1:59089/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1117782 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1117783 T3661 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1117784 T3661 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1117786 T3661 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1119644 T3661 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1119646 T3661 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1119649 T3661 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:64054
   [junit4]   2> 1119650 T3661 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1119650 T3661 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1119651 T3661 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003
   [junit4]   2> 1119651 T3661 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003/'
   [junit4]   2> 1119701 T3661 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003/solr.xml
   [junit4]   2> 1119816 T3661 oasc.CoreContainer.<init> New CoreContainer 1297865482
   [junit4]   2> 1119817 T3661 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003/]
   [junit4]   2> 1119819 T3661 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1119819 T3661 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1119820 T3661 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1119820 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1119821 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1119821 T3661 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1119822 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1119822 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1119823 T3661 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1119823 T3661 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1119825 T3661 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1119825 T3661 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1119826 T3661 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1119826 T3661 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1119827 T3661 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59089/solr
   [junit4]   2> 1119828 T3661 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1119828 T3661 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1119829 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1119831 T3712 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@531ecfcc name:ZooKeeperConnection Watcher:127.0.0.1:59089 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1119832 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1119833 T3661 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1119835 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1119836 T3715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3beb5ee8 name:ZooKeeperConnection Watcher:127.0.0.1:59089/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1119837 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1119842 T3661 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1120845 T3661 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64054_
   [junit4]   2> 1120846 T3661 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64054_
   [junit4]   2> 1120850 T3661 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1120855 T3661 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1120855 T3661 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1123192 T3661 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1123193 T3661 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1123197 T3661 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:11889
   [junit4]   2> 1123198 T3661 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1123198 T3661 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1123199 T3661 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004
   [junit4]   2> 1123199 T3661 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004/'
   [junit4]   2> 1123259 T3661 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004/solr.xml
   [junit4]   2> 1123367 T3661 oasc.CoreContainer.<init> New CoreContainer 267175162
   [junit4]   2> 1123368 T3661 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004/]
   [junit4]   2> 1123369 T3661 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1123369 T3661 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1123370 T3661 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1123370 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1123370 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1123371 T3661 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1123371 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1123371 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1123372 T3661 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1123372 T3661 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1123373 T3661 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1123373 T3661 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1123374 T3661 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1123374 T3661 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1123375 T3661 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59089/solr
   [junit4]   2> 1123375 T3661 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1123376 T3661 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1123377 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1123379 T3728 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5720d785 name:ZooKeeperConnection Watcher:127.0.0.1:59089 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1123380 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1123380 T3661 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1123383 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1123385 T3731 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c183b2b name:ZooKeeperConnection Watcher:127.0.0.1:59089/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1123385 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1123392 T3661 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1124397 T3661 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:11889_
   [junit4]   2> 1124398 T3661 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:11889_
   [junit4]   2> 1124403 T3661 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1124410 T3661 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1124411 T3661 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1126267 T3661 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1126269 T3661 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1126272 T3661 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:15642
   [junit4]   2> 1126273 T3661 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1126274 T3661 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1126274 T3661 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005
   [junit4]   2> 1126275 T3661 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005/'
   [junit4]   2> 1126345 T3661 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005/solr.xml
   [junit4]   2> 1126426 T3661 oasc.CoreContainer.<init> New CoreContainer 1627516725
   [junit4]   2> 1126426 T3661 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005/]
   [junit4]   2> 1126427 T3661 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1126428 T3661 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1126428 T3661 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1126428 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1126429 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1126429 T3661 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1126429 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1126430 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1126430 T3661 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1126431 T3661 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1126431 T3661 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1126432 T3661 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1126432 T3661 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1126432 T3661 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1126433 T3661 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59089/solr
   [junit4]   2> 1126433 T3661 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1126434 T3661 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1126435 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1126437 T3744 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17ef5d05 name:ZooKeeperConnection Watcher:127.0.0.1:59089 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1126437 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1126438 T3661 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1126439 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1126441 T3747 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7121b5c9 name:ZooKeeperConnection Watcher:127.0.0.1:59089/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1126441 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1126446 T3661 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1127449 T3661 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:15642_
   [junit4]   2> 1127451 T3661 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:15642_
   [junit4]   2> 1127458 T3661 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1127466 T3661 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1127467 T3661 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1129564 T3661 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1129565 T3661 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1129570 T3661 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:18971
   [junit4]   2> 1129571 T3661 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1129572 T3661 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1129572 T3661 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006
   [junit4]   2> 1129573 T3661 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006/'
   [junit4]   2> 1129644 T3661 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006/solr.xml
   [junit4]   2> 1129776 T3661 oasc.CoreContainer.<init> New CoreContainer 197005117
   [junit4]   2> 1129776 T3661 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006/]
   [junit4]   2> 1129778 T3661 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1129778 T3661 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1129778 T3661 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1129779 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1129779 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1129780 T3661 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1129780 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1129780 T3661 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1129781 T3661 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1129781 T3661 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1129782 T3661 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1129782 T3661 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1129782 T3661 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1129783 T3661 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1129783 T3661 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:59089/solr
   [junit4]   2> 1129784 T3661 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1129784 T3661 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1129785 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1129787 T3760 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52a418b7 name:ZooKeeperConnection Watcher:127.0.0.1:59089 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1129788 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1129788 T3661 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1129791 T3661 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1129793 T3763 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37d7eb6d name:ZooKeeperConnection Watcher:127.0.0.1:59089/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1129793 T3661 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1129800 T3661 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1130805 T3661 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:18971_
   [junit4]   2> 1130806 T3661 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:18971_
   [junit4]   2> 1130812 T3661 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1131236 T3661 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3
   [junit4]   2> 1131236 T3661 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1131253 T3707 oasha.CollectionsHandler.handleCreateAction Creating Collection : router.field=myOwnField&action=CREATE&version=2&collection.configName=conf1&replicationFactor=2&numShards=2&name=solrj_collection&wt=javabin
   [junit4]   2> 1131255 T3693 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/collection-queue-work state SyncConnected
   [junit4]   2> 1131259 T3695 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1131260 T3765 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1131262 T3765 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 1131263 T3765 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 1131269 T3693 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1131275 T3694 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1131275 T3694 oasc.Overseer$ClusterStateUpdater.buildCollection Building a new collection: solrj_collection
   [junit4]   2> 1131276 T3694 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 1131276 T3694 oasc.Overseer$ClusterStateUpdater.createCollection state version solrj_collection 1
   [junit4]   2> 1131280 T3693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131281 T3731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131281 T3700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131280 T3747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131281 T3715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131281 T3763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131370 T3765 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1131371 T3765 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:64054_
   [junit4]   2> 1131372 T3765 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:11889_
   [junit4]   2> 1131376 T3765 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:18971_
   [junit4]   2> 1131376 T3704 oasha.CoreAdminHandler.handleCreateAction core create command shard=shard1&action=CREATE&version=2&qt=/admin/cores&collection.configName=conf1&numShards=2&collection=solrj_collection&name=solrj_collection_shard1_replica1&wt=javabin
   [junit4]   2> 1131378 T3765 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:15642_
   [junit4]   2> 1131378 T3719 oasha.CoreAdminHandler.handleCreateAction core create command shard=shard1&action=CREATE&version=2&qt=/admin/cores&collection.configName=conf1&numShards=2&collection=solrj_collection&name=solrj_collection_shard1_replica2&wt=javabin
   [junit4]   2> 1131380 T3751 oasha.CoreAdminHandler.handleCreateAction core create command shard=shard2&action=CREATE&version=2&qt=/admin/cores&collection.configName=conf1&numShards=2&collection=solrj_collection&name=solrj_collection_shard2_replica1&wt=javabin
   [junit4]   2> 1131381 T3737 oasha.CoreAdminHandler.handleCreateAction core create command shard=shard2&action=CREATE&version=2&qt=/admin/cores&collection.configName=conf1&numShards=2&collection=solrj_collection&name=solrj_collection_shard2_replica2&wt=javabin
   [junit4]   2> 1131382 T3704 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 1131383 T3719 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 1131384 T3704 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1131384 T3693 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1131385 T3719 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1131386 T3737 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 1131388 T3737 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1131390 T3751 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 1131392 T3694 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64054",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:64054_",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 1131394 T3751 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1131397 T3694 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 3, message = {
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:11889",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:11889_",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 1131422 T3693 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1131422 T3694 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 3, message = {
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:15642",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:15642_",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 1131426 T3694 oasc.Overseer$ClusterStateUpdater.processMessage processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:18971",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "node_name":"127.0.0.1:18971_",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "state":"down"}
   [junit4]   2> 1131430 T3693 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1131435 T3763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131435 T3747 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131435 T3700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131435 T3693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131436 T3715 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1131439 T3731 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1132385 T3704 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 1132386 T3704 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1132386 T3719 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 1132387 T3719 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1132387 T3704 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1132388 T3704 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1132388 T3737 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 1132388 T3719 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1132389 T3737 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1132390 T3719 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1132391 T3737 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1132391 T3737 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1132392 T3719 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1132392 T3704 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1132393 T3704 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003/solrj_collection_shard1_replica1/'
   [junit4]   2> 1132393 T3737 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1132394 T3737 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005/solrj_collection_shard2_replica2/'
   [junit4]   2> 1132395 T3751 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 1132395 T3719 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004/solrj_collection_shard1_replica2/'
   [junit4]   2> 1132397 T3751 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1132399 T3751 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1132399 T3751 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1132404 T3751 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1132405 T3751 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006/solrj_collection_shard2_replica1/'
   [junit4]   2> 1132431 T3704 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1132438 T3719 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1132443 T3719 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1132448 T3737 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1132456 T3737 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1132457 T3751 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1132462 T3704 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1132463 T3751 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1132503 T3719 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1132520 T3737 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1132522 T3704 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1132535 T3751 oasc.SolrConfig.<init> Using Lucene MatchVersion: 5.0.0
   [junit4]   2> 1132585 T3719 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1132585 T3737 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1132588 T3704 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1132597 T3751 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1132686 T3719 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1132687 T3737 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1132689 T3704 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1132705 T3719 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 1132709 T3704 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 1132724 T3737 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 1132747 T3751 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1132760 T3751 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 1133414 T3737 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1133418 T3704 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1133431 T3737 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1133431 T3719 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1133432 T3704 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1133437 T3737 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1133438 T3704 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1133450 T3719 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1133455 T3719 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1133463 T3737 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1133463 T3704 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1133471 T3704 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1133471 T3737 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1133472 T3719 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1133479 T3704 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1133481 T3719 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1133484 T3737 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1133484 T3704 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1133486 T3704 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1133486 T3737 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1133486 T3704 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1133487 T3737 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1133488 T3737 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1133488 T3704 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1133488 T3704 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1133489 T3737 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1133489 T3704 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 1133489 T3737 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1133490 T3719 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1133490 T3737 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 1133489 T3704 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1133490 T3737 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1133491 T3704 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003/solrj_collection_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003/solrj_collection_shard1_replica1/data/
   [junit4]   2> 1133491 T3719 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1133491 T3704 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16ea6719
   [junit4]   2> 1133491 T3737 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005/solrj_collection_shard2_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005/solrj_collection_shard2_replica2/data/
   [junit4]   2> 1133492 T3719 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1133492 T3737 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16ea6719
   [junit4]   2> 1133493 T3704 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003/solrj_collection_shard1_replica1/data
   [junit4]   2> 1133493 T3719 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1133494 T3704 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 1133494 T3737 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005/solrj_collection_shard2_replica2/data
   [junit4]   2> 1133494 T3719 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1133494 T3704 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1133495 T3719 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1133496 T3704 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-003/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 1133495 T3737 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 1133496 T3704 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=62.1337890625, floorSegmentMB=0.69921875, forceMergeDeletesPctAllowed=25.151714242985573, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1133496 T3719 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 1133497 T3737 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1133498 T3704 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7cae3491 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@10fd5b8a),segFN=segments_1,generation=1}
   [junit4]   2> 1133498 T3737 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-005/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 1133498 T3704 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1133498 T3719 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1133499 T3737 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=62.1337890625, floorSegmentMB=0.69921875, forceMergeDeletesPctAllowed=25.151714242985573, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1133500 T3719 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004/solrj_collection_shard1_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004/solrj_collection_shard1_replica2/data/
   [junit4]   2> 1133501 T3719 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16ea6719
   [junit4]   2> 1133501 T3737 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@2e3851ff lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@738f540d),segFN=segments_1,generation=1}
   [junit4]   2> 1133503 T3719 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004/solrj_collection_shard1_replica2/data
   [junit4]   2> 1133503 T3737 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1133503 T3719 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 1133504 T3719 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1133505 T3719 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-004/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 1133506 T3719 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=62.1337890625, floorSegmentMB=0.69921875, forceMergeDeletesPctAllowed=25.151714242985573, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1133508 T3719 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3c83ba5 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4ec3cf93),segFN=segments_1,generation=1}
   [junit4]   2> 1133508 T3704 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1133508 T3719 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1133509 T3704 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1133510 T3704 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1133510 T3704 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1133511 T3704 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1133511 T3704 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1133512 T3737 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1133512 T3704 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1133512 T3737 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1133513 T3704 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1133513 T3737 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1133514 T3704 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1133514 T3737 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1133515 T3737 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1133515 T3704 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133516 T3737 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1133517 T3704 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133517 T3719 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1133517 T3737 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1133518 T3719 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1133518 T3704 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133519 T3719 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1133518 T3737 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1133520 T3704 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133520 T3719 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1133521 T3704 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1133521 T3719 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1133520 T3737 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1133522 T3704 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1133522 T3719 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1133523 T3737 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133524 T3719 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1133524 T3737 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133523 T3704 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 1133525 T3719 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1133526 T3751 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1133525 T3737 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133526 T3704 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1133526 T3719 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1133527 T3737 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133528 T3704 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1133529 T3719 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133529 T3737 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1133530 T3719 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133530 T3704 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1133530 T3737 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1133531 T3719 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133531 T3704 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1133532 T3737 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 1133532 T3719 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1133533 T3704 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1133533 T3737 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1133533 T3719 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1133534 T3704 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1133534 T3737 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1133535 T3719 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1133535 T3704 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1133536 T3719 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 1133537 T3751 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1133537 T3719 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1133537 T3704 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1133536 T3737 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1133539 T3704 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1133538 T3719 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1133539 T3737 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1133540 T3704 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1133541 T3737 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1133541 T3719 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1133541 T3751 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1133541 T3704 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1133542 T3719 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1133542 T3737 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1133543 T3704 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1133543 T3719 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1133544 T3737 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1133545 T3719 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1133545 T3737 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1133545 T3719 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1133546 T3737 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1133546 T3719 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1133547 T3737 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1133547 T3719 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1133548 T3737 oasc.RequestHandlers.initHandlersFromConfig created /adm

[...truncated too long message...]

arrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1343416 T3661 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1343417 T3661 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006/conf1_shard1_replica1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006/conf1_shard1_replica1/data;done=false>>]
   [junit4]   2> 1343417 T3661 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006/conf1_shard1_replica1/data
   [junit4]   2> 1343418 T3661 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006/conf1_shard1_replica1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006/conf1_shard1_replica1/data/index;done=false>>]
   [junit4]   2> 1343418 T3661 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001/tempDir-006/conf1_shard1_replica1/data/index
   [junit4]   2> 1343419 T3661 oasc.Overseer.close Overseer (id=92900272846012430-127.0.0.1:18971_-n_0000000004) closing
   [junit4]   2> 1343420 T3830 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:18971_
   [junit4]   2> 1343424 T3663 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x14a0c4e5286000e, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 1344924 T3828 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1344925 T3661 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 1349642 T3661 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 1349645 T3661 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59089 59089
   [junit4]   2> 1349752 T3662 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:59089 59089
   [junit4]   2> 1349754 T3662 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/clusterstate.json
   [junit4]   2> 		6	/solr/aliases.json
   [junit4]   2> 		5	/solr/configs/conf1
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/queue
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=19B2B8D85CD3422A -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=en_ZA -Dtests.timezone=Asia/Dacca -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    235s J3 | CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: reload the collection time out:180s
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([19B2B8D85CD3422A:985436C02B8C2216]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:569)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.request.CollectionAdminRequest.process(CollectionAdminRequest.java:379)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testSolrJAPICalls(CollectionsAPIDistributedZkTest.java:332)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:201)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1349778 T3661 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-19B2B8D85CD3422A-001
   [junit4]   2> 234683 T3660 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=DefaultSimilarity, locale=en_ZA, timezone=Asia/Dacca
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=134704832,total=336592896
   [junit4]   2> NOTE: All tests run in this JVM: [OutputWriterTest, BlockDirectoryTest, MergeStrategyTest, ZkCLITest, TestDistributedGrouping, StatelessScriptUpdateProcessorFactoryTest, DocValuesMultiTest, DocumentAnalysisRequestHandlerTest, TestHighlightDedupGrouping, TestSchemaManager, TestFastWriter, CoreAdminHandlerTest, TestInitParams, WordBreakSolrSpellCheckerTest, SOLR749Test, TestSolrConfigHandlerCloud, DistribCursorPagingTest, TestSolrConfigHandler, TestTolerantSearch, PreAnalyzedUpdateProcessorTest, HttpPartitionTest, AssignTest, TestSolrConfigHandlerConcurrent, SolrPluginUtilsTest, TestSort, QueryResultKeyTest, TestCoreDiscovery, TestMergePolicyConfig, SpellCheckComponentTest, TestBadConfig, VMParamsZkACLAndCredentialsProvidersTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J3 in 234.99s, 1 test, 1 error <<< FAILURES!

[...truncated 917 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:536: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:477: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/build.xml:190: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/solr/common-build.xml:508: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:1363: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-5.x/lucene/common-build.xml:970: There were test failures: 450 suites, 1844 tests, 1 error, 56 ignored (10 assumptions)

Total time: 173 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-5.x #689
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 478226271 bytes
Compression is 0.0%
Took 1 min 25 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure