You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/05/08 14:10:34 UTC

[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 560 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/560/

1 tests failed.
REGRESSION:  org.apache.lucene.index.TestBinaryDocValuesUpdates.testTonsOfUpdates

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([BB3C75CC6546BB18:C319ABC7876694FA]:0)
	at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:75)
	at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:48)
	at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:152)
	at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:138)
	at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
	at org.apache.lucene.util.packed.AbstractBlockPackedWriter.writeValues(AbstractBlockPackedWriter.java:130)
	at org.apache.lucene.util.packed.MonotonicBlockPackedWriter.flush(MonotonicBlockPackedWriter.java:92)
	at org.apache.lucene.util.packed.AbstractBlockPackedWriter.add(AbstractBlockPackedWriter.java:80)
	at org.apache.lucene.util.packed.MonotonicBlockPackedWriter.add(MonotonicBlockPackedWriter.java:69)
	at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addBinaryField(MemoryDocValuesConsumer.java:278)
	at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addBinaryField(PerFieldDocValuesFormat.java:111)
	at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:421)
	at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:282)
	at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3086)
	at org.apache.lucene.index.IndexWriter.applyDeletesAndPurge(IndexWriter.java:4559)
	at org.apache.lucene.index.DocumentsWriter$ApplyDeletesEvent.process(DocumentsWriter.java:686)
	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4593)
	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4585)
	at org.apache.lucene.index.IndexWriter.updateBinaryDocValue(IndexWriter.java:1521)
	at org.apache.lucene.index.TestBinaryDocValuesUpdates.testTonsOfUpdates(TestBinaryDocValuesUpdates.java:1360)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)




Build Log:
[...truncated 1547 lines...]
   [junit4] Suite: org.apache.lucene.index.TestBinaryDocValuesUpdates
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBinaryDocValuesUpdates -Dtests.method=testTonsOfUpdates -Dtests.seed=BB3C75CC6546BB18 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_HN -Dtests.timezone=Etc/GMT+6 -Dtests.file.encoding=UTF-8
   [junit4] ERROR    164s J0 | TestBinaryDocValuesUpdates.testTonsOfUpdates <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BB3C75CC6546BB18:C319ABC7876694FA]:0)
   [junit4]    > 	at org.apache.lucene.store.RAMFile.newBuffer(RAMFile.java:75)
   [junit4]    > 	at org.apache.lucene.store.RAMFile.addBuffer(RAMFile.java:48)
   [junit4]    > 	at org.apache.lucene.store.RAMOutputStream.switchCurrentBuffer(RAMOutputStream.java:152)
   [junit4]    > 	at org.apache.lucene.store.RAMOutputStream.writeBytes(RAMOutputStream.java:138)
   [junit4]    > 	at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
   [junit4]    > 	at org.apache.lucene.util.packed.AbstractBlockPackedWriter.writeValues(AbstractBlockPackedWriter.java:130)
   [junit4]    > 	at org.apache.lucene.util.packed.MonotonicBlockPackedWriter.flush(MonotonicBlockPackedWriter.java:92)
   [junit4]    > 	at org.apache.lucene.util.packed.AbstractBlockPackedWriter.add(AbstractBlockPackedWriter.java:80)
   [junit4]    > 	at org.apache.lucene.util.packed.MonotonicBlockPackedWriter.add(MonotonicBlockPackedWriter.java:69)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryDocValuesConsumer.addBinaryField(MemoryDocValuesConsumer.java:278)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addBinaryField(PerFieldDocValuesFormat.java:111)
   [junit4]    > 	at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:421)
   [junit4]    > 	at org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:282)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3086)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.applyDeletesAndPurge(IndexWriter.java:4559)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter$ApplyDeletesEvent.process(DocumentsWriter.java:686)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4593)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:4585)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateBinaryDocValue(IndexWriter.java:1521)
   [junit4]    > 	at org.apache.lucene.index.TestBinaryDocValuesUpdates.testTonsOfUpdates(TestBinaryDocValuesUpdates.java:1360)
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/core/test/J0/./lucene.index.TestBinaryDocValuesUpdates-BB3C75CC6546BB18-001
   [junit4]   2> NOTE: test params are: codec=Lucene46: {f=PostingsFormat(name=Direct), foo=PostingsFormat(name=Direct), k2=PostingsFormat(name=Memory doPackFST= true), key=PostingsFormat(name=Memory doPackFST= true), id=PostingsFormat(name=Memory doPackFST= true), updKey=PostingsFormat(name=Asserting), upd=PostingsFormat(name=NestedPulsing), k1=PostingsFormat(name=Direct), dvUpdateKey=PostingsFormat(name=Direct), bdv=PostingsFormat(name=Asserting)}, docValues:{f2=DocValuesFormat(name=Disk), ndv=DocValuesFormat(name=Disk), sdv=DocValuesFormat(name=SimpleText), cf9=DocValuesFormat(name=Memory), f16=DocValuesFormat(name=Asserting), cf5=DocValuesFormat(name=Memory), cf8=DocValuesFormat(name=SimpleText), f7=DocValuesFormat(name=SimpleText), f3=DocValuesFormat(name=SimpleText), f15=DocValuesFormat(name=Memory), cf7=DocValuesFormat(name=Disk), cf15=DocValuesFormat(name=Asserting), control=DocValuesFormat(name=SimpleText), f11=DocValuesFormat(name=Memory), cf4=DocValuesFormat(name=SimpleText), cf1=DocValuesFormat(name=Memory), bdv=DocValuesFormat(name=Disk), f23=DocValuesFormat(name=Asserting), f24=DocValuesFormat(name=Disk), cf2=DocValuesFormat(name=Asserting), cf3=DocValuesFormat(name=Disk), f19=DocValuesFormat(name=Memory), f0=DocValuesFormat(name=Memory), cf16=DocValuesFormat(name=Disk), cf26=DocValuesFormat(name=Asserting), cf10=DocValuesFormat(name=Memory), val=DocValuesFormat(name=SimpleText), cf0=DocValuesFormat(name=SimpleText), cf21=DocValuesFormat(name=Memory), f22=DocValuesFormat(name=Memory), f21=DocValuesFormat(name=SimpleText), f26=DocValuesFormat(name=Memory), cf22=DocValuesFormat(name=Asserting), f6=DocValuesFormat(name=Disk), f10=DocValuesFormat(name=SimpleText), f14=DocValuesFormat(name=SimpleText), f8=DocValuesFormat(name=Memory), f20=DocValuesFormat(name=Disk), cf23=DocValuesFormat(name=Disk), f1=DocValuesFormat(name=Asserting), cf18=DocValuesFormat(name=Memory), f4=DocValuesFormat(name=Memory), cf24=DocValuesFormat(name=SimpleText), cf14=DocValuesFormat(name=Memory), cf11=DocValuesFormat(name=Asserting), cf25=DocValuesFormat(name=Memory), cf12=DocValuesFormat(name=Disk), f18=DocValuesFormat(name=SimpleText), cf=DocValuesFormat(name=Asserting), f9=DocValuesFormat(name=Asserting), f17=DocValuesFormat(name=Disk), bdv1=DocValuesFormat(name=SimpleText), cf17=DocValuesFormat(name=SimpleText), ssdv=DocValuesFormat(name=Memory), cf6=DocValuesFormat(name=Asserting), f25=DocValuesFormat(name=SimpleText), f12=DocValuesFormat(name=Asserting), f=DocValuesFormat(name=Memory), bdv2=DocValuesFormat(name=Memory), f5=DocValuesFormat(name=Asserting), f13=DocValuesFormat(name=Disk), cf13=DocValuesFormat(name=SimpleText), cf20=DocValuesFormat(name=SimpleText), cf19=DocValuesFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_HN, timezone=Etc/GMT+6
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=112441248,total=533069824
   [junit4]   2> NOTE: All tests run in this JVM: [TestDocIdBitSet, TestShardSearching, TestSimilarity, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, Nested, TestFieldCacheTermsFilter, TestIndexWriterReader, TestSloppyPhraseQuery2, TestSearcherManager, TestFieldCacheRewriteMethod, TestFieldCacheSanityChecker, TestTopDocsMerge, TestStoredFieldsFormat, TestDocumentsWriterStallControl, TestMultiLevelSkipList, TestRecyclingIntBlockAllocator, TestElevationComparator, TestDocTermOrds, TestSloppyMath, TestRollback, TestMergeSchedulerExternal, TestDisjunctionMaxQuery, TestFastDecompressionMode, TestPositiveScoresOnlyCollector, TestIndexCommit, TestSameTokenSamePosition, TestFlushByRamOrCountsPolicy, TestCompoundFile, TestNumericDocValuesUpdates, TestLock, TestBlockPostingsFormat3, TestCompiledAutomaton, TestCachingWrapperFilter, TestIndexWriterMerging, TestDemo, TestSearchForDuplicates, TestBroadWord, TestIndexWriter, TestLucene40PostingsFormat, TestUTF32ToUTF8, TestMultiTermConstantScore, TestNRTThreads, TestNumericTokenStream, TestTransactionRollback, TestLookaheadTokenFilter, TestMockCharFilter, TestFileSwitchDirectory, TestOfflineSorter, TestScoreCachingWrappingScorer, TestIndexWriterOutOfFileDescriptors, TestRollingBuffer, TestComplexExplanationsOfNonMatches, TestVersion, TestSpanFirstQuery, TestMaxTermFrequency, TestLucene40DocValuesFormat, TestMockAnalyzer, TestStressIndexing, TestDocTermOrdsRewriteMethod, TestNumericRangeQuery32, TestPrefixRandom, TestDirectoryReader, TestCharTermAttributeImpl, TestVirtualMethod, TestPhrasePrefixQuery, TestSizeBoundedForceMerge, TestTimeLimitingCollector, TestConcurrentMergeScheduler, TestFilterIterator, TestLucene42DocValuesFormat, TestLucene40PostingsReader, TestPhraseQuery, TestBagOfPositions, TestIndexWriterOutOfMemory, TestDocValuesWithThreads, TestComplexExplanations, TestReuseDocsEnum, TestRamUsageEstimator, Test2BNumericDocValues, Test2BPagedBytes, TestSearchAfter, TestNoMergePolicy, TestSegmentTermDocs, TestSort, TestTermVectorsWriter, TestCustomSearcherSort, TestIntBlockPool, TestIndexWriterOnJRECrash, TestEarlyTermination, TestSentinelIntSet, TestTermRangeQuery, TestBytesRefHash, TestStressIndexing2, TestAutomatonQueryUnicode, TestAllFilesHaveCodecHeader, TestSegmentReader, TestTopFieldCollector, TestWAH8DocIdSet, TestSetOnce, TestBinaryDocValuesUpdates]
   [junit4] Completed on J0 in 179.72s, 32 tests, 1 error <<< FAILURES!

[...truncated 103 lines...]
   [junit4] JVM J0: stdout was not empty, see: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/core/test/temp/junit4-J0-20140508_110457_197.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/heapdumps/java_pid15705.hprof ...
   [junit4] Heap dump file created [524288070 bytes in 1.995 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:49: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1296: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:920: There were test failures: 396 suites, 3083 tests, 1 error, 105 ignored (95 assumptions)

Total time: 64 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 563 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/563/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
No registered leader was found after waiting for 60000ms , collection: c8n_1x3_lf slice: shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found after waiting for 60000ms , collection: c8n_1x3_lf slice: shard1
	at __randomizedtesting.SeedInfo.seed([33447504DD538931:B2A2FB1CAA0CE90D]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:567)
	at org.apache.solr.cloud.HttpPartitionTest.testRf3WithLeaderFailover(HttpPartitionTest.java:349)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:148)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 11383 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/init-core-data-001
   [junit4]   2> 959965 T4540 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /sg/n
   [junit4]   2> 959971 T4540 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 959972 T4540 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 959973 T4541 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 960074 T4540 oasc.ZkTestServer.run start zk server on port:35002
   [junit4]   2> 960075 T4540 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 960081 T4547 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@576d60d7 name:ZooKeeperConnection Watcher:127.0.0.1:35002 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 960081 T4540 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 960082 T4540 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 960099 T4540 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 960101 T4549 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d6139e9 name:ZooKeeperConnection Watcher:127.0.0.1:35002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 960101 T4540 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 960102 T4540 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 960113 T4540 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 960115 T4540 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 960116 T4540 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 960126 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 960127 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 960139 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 960140 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 960244 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 960245 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 960256 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 960257 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 960260 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 960260 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 960263 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 960264 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 960266 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 960267 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 960270 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 960271 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 960273 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 960274 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 960281 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 960282 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 960284 T4540 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 960285 T4540 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 961578 T4540 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 961583 T4540 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:35006
   [junit4]   2> 961584 T4540 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 961584 T4540 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 961584 T4540 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-002
   [junit4]   2> 961585 T4540 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-002/'
   [junit4]   2> 961616 T4540 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-002/solr.xml
   [junit4]   2> 961688 T4540 oasc.CoreContainer.<init> New CoreContainer 860622507
   [junit4]   2> 961689 T4540 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-002/]
   [junit4]   2> 961690 T4540 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 961691 T4540 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 961691 T4540 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 961691 T4540 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 961692 T4540 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 961692 T4540 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 961692 T4540 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 961692 T4540 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 961693 T4540 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 961693 T4540 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 961694 T4540 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 961694 T4540 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 961694 T4540 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35002/solr
   [junit4]   2> 961695 T4540 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 961695 T4540 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 961699 T4560 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@465716dc name:ZooKeeperConnection Watcher:127.0.0.1:35002 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 961701 T4540 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 961726 T4540 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 961728 T4562 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@335f545b name:ZooKeeperConnection Watcher:127.0.0.1:35002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 961728 T4540 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 961740 T4540 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 961751 T4540 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 961754 T4540 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 961765 T4540 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 961767 T4540 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 961778 T4540 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 961780 T4540 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35005_sg%2Fn
   [junit4]   2> 961781 T4540 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35005_sg%2Fn
   [junit4]   2> 961791 T4540 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 961792 T4540 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 961803 T4540 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:35005_sg%2Fn
   [junit4]   2> 961804 T4540 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 961806 T4540 oasc.Overseer.start Overseer (id=91821773135478787-127.0.0.1:35005_sg%2Fn-n_0000000000) starting
   [junit4]   2> 961815 T4540 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 961831 T4564 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 961831 T4540 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 961835 T4540 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 961836 T4540 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 961838 T4563 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 961844 T4565 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 961845 T4565 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 961845 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 961845 T4565 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 961846 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 961847 T4563 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35005/sg/n",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35005_sg%2Fn",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 961847 T4563 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 961847 T4563 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 961849 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 961850 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 962848 T4565 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 962848 T4565 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 962849 T4565 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 962849 T4565 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 962850 T4565 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 962850 T4565 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-002/collection1/'
   [junit4]   2> 962852 T4565 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 962853 T4565 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 962853 T4565 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 962937 T4565 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 963015 T4565 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 963117 T4565 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 963128 T4565 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 963741 T4565 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 963758 T4565 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 963762 T4565 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 963772 T4565 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 963777 T4565 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 963781 T4565 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 963781 T4565 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 963782 T4565 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 963782 T4565 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 963783 T4565 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 963783 T4565 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 963783 T4565 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 963783 T4565 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 963784 T4565 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-002/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/control/data/
   [junit4]   2> 963784 T4565 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6decf9a6
   [junit4]   2> 963785 T4565 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/control/data
   [junit4]   2> 963785 T4565 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/control/data/index/
   [junit4]   2> 963785 T4565 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 963786 T4565 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/control/data/index
   [junit4]   2> 963786 T4565 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=4, maxMergeAtOnceExplicit=2, maxMergedSegmentMB=28.6474609375, floorSegmentMB=0.2548828125, forceMergeDeletesPctAllowed=23.993199250350468, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 963787 T4565 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@72cfa03c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@334b9aea),segFN=segments_1,generation=1}
   [junit4]   2> 963787 T4565 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 963791 T4565 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 963791 T4565 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 963791 T4565 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 963792 T4565 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 963792 T4565 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 963792 T4565 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 963793 T4565 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 963793 T4565 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 963793 T4565 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 963794 T4565 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 963794 T4565 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 963794 T4565 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 963795 T4565 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 963795 T4565 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 963796 T4565 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 963796 T4565 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 963797 T4565 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 963818 T4565 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 963825 T4565 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 963825 T4565 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 963826 T4565 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=498083169, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 963827 T4565 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@72cfa03c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@334b9aea),segFN=segments_1,generation=1}
   [junit4]   2> 963827 T4565 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 963827 T4565 oass.SolrIndexSearcher.<init> Opening Searcher@68ed15d8[collection1] main
   [junit4]   2> 963828 T4565 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 963829 T4565 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 963829 T4565 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 963829 T4565 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 963829 T4565 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 963830 T4565 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 963830 T4565 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 963830 T4565 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 963831 T4565 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 963831 T4565 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 963833 T4566 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@68ed15d8[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 963834 T4565 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 963835 T4569 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35005/sg/n collection:control_collection shard:shard1
   [junit4]   2> 963835 T4540 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 963836 T4540 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 963836 T4569 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 963840 T4540 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 963851 T4572 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fc1f302 name:ZooKeeperConnection Watcher:127.0.0.1:35002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 963851 T4540 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 963853 T4540 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 963854 T4569 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 963865 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 963865 T4540 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 963866 T4569 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 963866 T4569 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 963866 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C1773 name=collection1 org.apache.solr.core.SolrCore@492ebf93 url=http://127.0.0.1:35005/sg/n/collection1 node=127.0.0.1:35005_sg%2Fn C1773_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:35005/sg/n, core=collection1, node_name=127.0.0.1:35005_sg%2Fn}
   [junit4]   2> 963866 T4569 C1773 P35005 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35005/sg/n/collection1/
   [junit4]   2> 963867 T4569 C1773 P35005 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 963867 T4569 C1773 P35005 oasc.SyncStrategy.syncToMe http://127.0.0.1:35005/sg/n/collection1/ has no replicas
   [junit4]   2> 963867 T4569 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35005/sg/n/collection1/ shard1
   [junit4]   2> 963868 T4569 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 963873 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 963884 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 963884 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 963898 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 963901 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 964003 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 964003 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 964052 T4569 oasc.ZkController.register We are http://127.0.0.1:35005/sg/n/collection1/ and leader is http://127.0.0.1:35005/sg/n/collection1/
   [junit4]   2> 964052 T4569 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35005/sg/n
   [junit4]   2> 964052 T4569 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 964052 T4569 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 964052 T4569 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 964054 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 964054 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 964054 T4569 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 964054 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 964055 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 964055 T4563 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35005/sg/n",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35005_sg%2Fn",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 964062 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 964166 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 964166 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 965043 T4540 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 965044 T4540 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 965048 T4540 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:35011
   [junit4]   2> 965049 T4540 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 965049 T4540 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 965049 T4540 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003
   [junit4]   2> 965050 T4540 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/'
   [junit4]   2> 965080 T4540 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/solr.xml
   [junit4]   2> 965153 T4540 oasc.CoreContainer.<init> New CoreContainer 1717051528
   [junit4]   2> 965154 T4540 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/]
   [junit4]   2> 965155 T4540 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 965155 T4540 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 965156 T4540 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 965156 T4540 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 965156 T4540 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 965157 T4540 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 965157 T4540 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 965157 T4540 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 965157 T4540 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 965158 T4540 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 965158 T4540 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 965159 T4540 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 965159 T4540 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35002/solr
   [junit4]   2> 965159 T4540 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 965160 T4540 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 965188 T4583 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23bff536 name:ZooKeeperConnection Watcher:127.0.0.1:35002 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 965190 T4540 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 965195 T4540 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 965197 T4585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a8adb3f name:ZooKeeperConnection Watcher:127.0.0.1:35002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 965197 T4540 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 965220 T4540 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 966222 T4540 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35010_sg%2Fn
   [junit4]   2> 966252 T4540 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35010_sg%2Fn
   [junit4]   2> 966343 T4586 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 966343 T4586 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 966363 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 966363 T4586 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 966363 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 966363 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 966364 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 966365 T4563 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35010/sg/n",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35010_sg%2Fn",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 966365 T4563 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 966365 T4563 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 966368 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 966409 T4585 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 966409 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 966409 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 967364 T4586 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 967364 T4586 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 967365 T4586 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 967365 T4586 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 967366 T4586 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 967366 T4586 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/collection1/'
   [junit4]   2> 967368 T4586 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 967368 T4586 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 967369 T4586 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 967464 T4586 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 967540 T4586 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 967642 T4586 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 967652 T4586 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 968241 T4586 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 968251 T4586 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 968253 T4586 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 968262 T4586 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 968265 T4586 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 968268 T4586 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 968269 T4586 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 968270 T4586 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 968271 T4586 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 968271 T4586 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 968272 T4586 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 968272 T4586 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 968272 T4586 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 968272 T4586 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/jetty1/
   [junit4]   2> 968272 T4586 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6decf9a6
   [junit4]   2> 968273 T4586 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/jetty1
   [junit4]   2> 968273 T4586 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/jetty1/index/
   [junit4]   2> 968274 T4586 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 968274 T4586 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/jetty1/index
   [junit4]   2> 968274 T4586 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=4, maxMergeAtOnceExplicit=2, maxMergedSegmentMB=28.6474609375, floorSegmentMB=0.2548828125, forceMergeDeletesPctAllowed=23.993199250350468, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 968275 T4586 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7e67f7eb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3933677a),segFN=segments_1,generation=1}
   [junit4]   2> 968275 T4586 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 968278 T4586 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 968278 T4586 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 968278 T4586 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 968279 T4586 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 968279 T4586 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 968279 T4586 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 968279 T4586 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 968280 T4586 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 968280 T4586 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 968280 T4586 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 968281 T4586 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 968281 T4586 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 968281 T4586 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 968281 T4586 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 968282 T4586 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 968282 T4586 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 968283 T4586 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 968294 T4586 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 968299 T4586 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 968299 T4586 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 968301 T4586 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=498083169, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 968302 T4586 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@7e67f7eb lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3933677a),segFN=segments_1,generation=1}
   [junit4]   2> 968302 T4586 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 968303 T4586 oass.SolrIndexSearcher.<init> Opening Searcher@5c4955dc[collection1] main
   [junit4]   2> 968303 T4586 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 968304 T4586 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 968304 T4586 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 968304 T4586 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 968304 T4586 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 968304 T4586 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 968305 T4586 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 968305 T4586 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 968305 T4586 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 968305 T4586 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 968308 T4587 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5c4955dc[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 968310 T4586 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 968310 T4590 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35010/sg/n collection:collection1 shard:shard2
   [junit4]   2> 968311 T4540 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 968312 T4540 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 968311 T4590 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 968333 T4590 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 968336 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 968336 T4590 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 968336 T4590 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1774 name=collection1 org.apache.solr.core.SolrCore@2e333c14 url=http://127.0.0.1:35010/sg/n/collection1 node=127.0.0.1:35010_sg%2Fn C1774_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:35010/sg/n, core=collection1, node_name=127.0.0.1:35010_sg%2Fn}
   [junit4]   2> 968336 T4590 C1774 P35010 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35010/sg/n/collection1/
   [junit4]   2> 968337 T4590 C1774 P35010 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 968337 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 968337 T4590 C1774 P35010 oasc.SyncStrategy.syncToMe http://127.0.0.1:35010/sg/n/collection1/ has no replicas
   [junit4]   2> 968337 T4590 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35010/sg/n/collection1/ shard2
   [junit4]   2> 968337 T4590 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 968346 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 968355 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 968355 T4585 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 968355 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 968370 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 968379 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 968482 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 968482 T4585 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 968482 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 968524 T4590 oasc.ZkController.register We are http://127.0.0.1:35010/sg/n/collection1/ and leader is http://127.0.0.1:35010/sg/n/collection1/
   [junit4]   2> 968524 T4590 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35010/sg/n
   [junit4]   2> 968524 T4590 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 968524 T4590 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 968524 T4590 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 968525 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 968526 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 968526 T4590 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 968526 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 968526 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 968527 T4563 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35010/sg/n",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35010_sg%2Fn",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 968537 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 968639 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 968639 T4585 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 968639 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 969502 T4540 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 969503 T4540 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 969506 T4540 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:35015
   [junit4]   2> 969507 T4540 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 969507 T4540 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 969508 T4540 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-004
   [junit4]   2> 969508 T4540 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-004/'
   [junit4]   2> 969545 T4540 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-004/solr.xml
   [junit4]   2> 969636 T4540 oasc.CoreContainer.<init> New CoreContainer 528936883
   [junit4]   2> 969637 T4540 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-004/]
   [junit4]   2> 969638 T4540 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 969638 T4540 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 969638 T4540 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 969639 T4540 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 969639 T4540 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 969639 T4540 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 969640 T4540 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 969640 T4540 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 969640 T4540 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 969641 T4540 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 969641 T4540 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 969641 T4540 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 969642 T4540 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35002/solr
   [junit4]   2> 969642 T4540 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 969643 T4540 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 969645 T4602 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53e2e0f0 name:ZooKeeperConnection Watcher:127.0.0.1:35002 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 969647 T4540 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 969656 T4540 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 969657 T4604 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30c8c486 name:ZooKeeperConnection Watcher:127.0.0.1:35002/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 969658 T4540 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 969680 T4540 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 970683 T4540 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35014_sg%2Fn
   [junit4]   2> 970691 T4540 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35014_sg%2Fn
   [junit4]   2> 970714 T4605 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 970714 T4605 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 970715 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 970716 T4605 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 970716 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 970716 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 970717 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 970718 T4563 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35014/sg/n",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35014_sg%2Fn",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 970718 T4563 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 970718 T4563 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 970720 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 970721 T4604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 970721 T4585 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 970721 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 970721 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 971718 T4605 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 971718 T4605 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 971719 T4605 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 971719 T4605 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 971720 T4605 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 971720 T4605 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-004/collection1/'
   [junit4]   2> 971722 T4605 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 971722 T4605 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 971723 T4605 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 971806 T4605 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 971902 T4605 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 972004 T4605 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 972015 T4605 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 972634 T4605 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 972648 T4605 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 972653 T4605 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 972664 T4605 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 972669 T4605 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 972673 T4605 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 972674 T4605 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 972676 T4605 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 972676 T4605 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 972677 T4605 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 972677 T4605 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 972677 T4605 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 972678 T4605 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 972678 T4605 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-004/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/jetty2/
   [junit4]   2> 972678 T4605 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6decf9a6
   [junit4]   2> 972680 T4605 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/jetty2
   [junit4]   2> 972681 T4605 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/jetty2/index/
   [junit4]   2> 972681 T4605 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 972682 T4605 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-001/jetty2/index
   [junit4]   2> 972682 T4605 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=4, maxMergeAtOnceExplicit=2, maxMergedSegmentMB=28.6474609375, floorSegmentMB=0.2548828125, forceMergeDeletesPctAllowed=23.993199250350468, segmentsPerTier=33.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 972683 T4605 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1246d3db lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@33bd1178),segFN=segments_1,generation=1}
   [junit4]   2> 972684 T4605 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 972688 T4605 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 972688 T4605 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 972688 T4605 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 972689 T4605 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 972689 T4605 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 972689 T4605 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 972690 T4605 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 972690 T4605 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 972691 T4605 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 972691 T4605 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 972692 T4605 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 972692 T4605 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 972692 T4605 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 972693 T4605 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 972693 T4605 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 972694 T4605 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 972695 T4605 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 972717 T4605 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 972723 T4605 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 972724 T4605 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 972725 T4605 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=498083169, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 972725 T4605 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1246d3db lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@33bd1178),segFN=segments_1,generation=1}
   [junit4]   2> 972726 T4605 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 972726 T4605 oass.SolrIndexSearcher.<init> Opening Searcher@144f2510[collection1] main
   [junit4]   2> 972726 T4605 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 972727 T4605 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 972727 T4605 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 972728 T4605 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 972728 T4605 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 972729 T4605 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 972729 T4605 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 972729 T4605 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 972730 T4605 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 972730 T4605 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 972733 T4606 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@144f2510[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 972736 T4605 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 972737 T4609 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35014/sg/n collection:collection1 shard:shard1
   [junit4]   2> 972737 T4540 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 972738 T4540 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 972737 T4609 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 972739 T4540 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 972741 T4540 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30000 for each attempt
   [junit4]   2> 972741 T4540 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
   [junit4]   2> 972742 T4540 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 972752 T4609 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 972760 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 972760 T4609 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 972760 T4609 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1775 name=collection1 org.apache.solr.core.SolrCore@7b9bb09 url=http://127.0.0.1:35014/sg/n/collection1 node=127.0.0.1:35014_sg%2Fn C1775_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:35014/sg/n, core=collection1, node_name=127.0.0.1:35014_sg%2Fn}
   [junit4]   2> 972760 T4609 C1775 P35014 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35014/sg/n/collection1/
   [junit4]   2> 972761 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 972761 T4609 C1775 P35014 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 972761 T4609 C1775 P35014 oasc.SyncStrategy.syncToMe http://127.0.0.1:35014/sg/n/collection1/ has no replicas
   [junit4]   2> 972761 T4609 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35014/sg/n/collection1/ shard1
   [junit4]   2> 972762 T4609 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 972769 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 972779 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 972779 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 972779 T4604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 972780 T4585 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 972792 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 972794 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 972897 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 972897 T4585 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 972897 T4604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 972897 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 972945 T4609 oasc.ZkController.register We are http://127.0.0.1:35014/sg/n/collection1/ and leader is http://127.0.0.1:35014/sg/n/collection1/
   [junit4]   2> 972945 T4609 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35014/sg/n
   [junit4]   2> 972945 T4609 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 972945 T4609 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 972945 T4609 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 972947 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 972947 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 972947 T4609 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 972947 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 972949 T4563 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 972949 T4563 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35014/sg/n",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:35014_sg%2Fn",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 972952 T4562 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 973054 T4562 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 973054 T4585 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 973054 T4604 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 973054 T4572 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 973752 T4540 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 973753 T4540 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
   [junit4]   2> 973755 T4570 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=35018,localport=35005], receiveBufferSize:6565888
   [junit4]   2> 973759 T4570 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=35006,localport=35019], receiveBufferSize=6565888
   [junit4]   2> ASYNC  NEW_CORE C1776 name=collection1 org.apache.solr.core.SolrCore@492ebf93 url=http://127.0.0.1:35005/sg/n/collection1 node=127.0.0.1:35005_sg%2Fn C1776_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:35005/sg/n, core=collection1, node_name=127.0.0.1:35005_sg%2Fn, leader=true}
   [junit4]   2> 973759 T4552 C1776 P35005 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 973760 T4552 C1776 P35005 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 973760 T4552 C1776 P35005 oasc.SolrCore.openNewSearcher SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 973761 T4552 C1776 P35005 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 973761 T4552 C1776 P35005 oasup.LogUpdateProcessor.finish [collection1] webapp=/sg/n path=/update params={version=2&wt=javabin&waitSearcher=true&softCommit=false&commit=true} {commit=} 0 2
   [junit4]   2> 973864 T4610 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=35020,localport=35014], receiveBufferSize:6565888
   [junit4]   2> 973867 T4610 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=35015,localport=35021], receiveBufferSize=6565888
   [junit4]   2> 973870 T4610 oasc.SocketProxy$Acceptor.run accepted Socket[addr=/140.211.11.196,port=35022,localport=35014], receiveBufferSize:6565888
   [junit4]   2> 973871 T4610 oasc.SocketProxy$Bridge.<init> proxy connection Socket[addr=/127.0.0.1,port=35015,localport=35023], receiveBufferSize=6565888
   [junit4]   2>  C1775_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:35014/sg/n, core=collection1, node_name=127.0.0.1:35014_sg%2Fn, leader=true}
   [junit4]   2> 973871 T4595 C1775 P35014 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 973872 T4595 C1775 P35014 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commi

[...truncated too long message...]

8n_1x3_lf_shard1_replica2] Closing main searcher on request.
   [junit4]   2> 1683199 T4585 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1683199 T4585 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/c8n_1x3_lf_shard1_replica2/data [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/c8n_1x3_lf_shard1_replica2/data;done=false>>]
   [junit4]   2> 1683199 T4585 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/c8n_1x3_lf_shard1_replica2/data
   [junit4]   2> 1683200 T4585 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/c8n_1x3_lf_shard1_replica2/data/index.20140526033211002 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/c8n_1x3_lf_shard1_replica2/data/index.20140526033211002;done=false>>]
   [junit4]   2> 1683200 T4585 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001/tempDir-003/c8n_1x3_lf_shard1_replica2/data/index.20140526033211002
   [junit4]   2> 1683200 T4585 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1683201 T4585 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1683201 T4585 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/c8n_1x3/leader_elect/shard1/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 1683201 T4585 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 1683202 T4585 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 1683233 T4540 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 35014
   [junit4]   2> 1683233 T4540 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 1683235 T4540 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:35002 35002
   [junit4]   2> 1683411 T4541 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:35002 35002
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HttpPartitionTest -Dtests.method=testDistribSearch -Dtests.seed=33447504DD538931 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=es -Dtests.timezone=America/New_York -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    723s J1 | HttpPartitionTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: No registered leader was found after waiting for 60000ms , collection: c8n_1x3_lf slice: shard1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([33447504DD538931:B2A2FB1CAA0CE90D]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:567)
   [junit4]    > 	at org.apache.solr.cloud.HttpPartitionTest.testRf3WithLeaderFailover(HttpPartitionTest.java:349)
   [junit4]    > 	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:148)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 1683426 T4540 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.HttpPartitionTest-33447504DD538931-001
   [junit4]   2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=es, timezone=America/New_York
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=100876376,total=521601024
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsDirectoryTest, TermVectorComponentDistributedTest, TestStressLucene, FunctionTest, SpellCheckCollatorTest, TestCustomSort, BasicDistributedZk2Test, CurrencyFieldXmlFileTest, TestBinaryField, OutputWriterTest, ZkNodePropsTest, TestAnalyzedSuggestions, SuggestComponentTest, HdfsBasicDistributedZkTest, ChaosMonkeySafeLeaderTest, DistanceFunctionTest, TestHashPartitioner, SystemInfoHandlerTest, HdfsLockFactoryTest, DocValuesMultiTest, TestCSVLoader, TestCollationField, TestLMDirichletSimilarityFactory, DebugComponentTest, SolrCoreCheckLockOnStartupTest, DateFieldTest, TestSchemaResource, CachingDirectoryFactoryTest, CurrencyFieldOpenExchangeTest, CollectionsAPIDistributedZkTest, DocValuesMissingTest, TestFastOutputStream, DateMathParserTest, AssignTest, HdfsCollectionsAPIDistributedZkTest, DeleteInactiveReplicaTest, TestSolrCoreProperties, TestSolrDeletionPolicy2, TestWordDelimiterFilterFactory, LegacyHTMLStripCharFilterTest, TestSolrJ, DirectUpdateHandlerTest, TestStressUserVersions, ResourceLoaderTest, HttpPartitionTest]
   [junit4] Completed on J1 in 723.51s, 1 test, 1 error <<< FAILURES!

[...truncated 930 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1296: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:920: There were test failures: 406 suites, 1688 tests, 1 error, 34 ignored (5 assumptions)

Total time: 222 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 562 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/562/

1 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=10163, name=qtp2051886554-10163, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10163, name=qtp2051886554-10163, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at __randomizedtesting.SeedInfo.seed([51907E065F3FC472]:0)
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:693)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1047)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
	at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:665)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 11953 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/init-core-data-001
   [junit4]   2> 1736948 T4075 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1736948 T4075 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /nm_hb/el
   [junit4]   2> 1736955 T4075 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1736956 T4075 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1736956 T4076 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1737057 T4075 oasc.ZkTestServer.run start zk server on port:48841
   [junit4]   2> 1737058 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1737062 T4082 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49d42f4 name:ZooKeeperConnection Watcher:127.0.0.1:48841 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1737063 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1737063 T4075 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1737077 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1737078 T4084 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@732f5bcf name:ZooKeeperConnection Watcher:127.0.0.1:48841/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1737079 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1737079 T4075 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1737084 T4075 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1737087 T4075 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1737096 T4075 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1737099 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1737099 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1737106 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1737107 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1737210 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1737211 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1737214 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1737215 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1737219 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1737219 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1737222 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1737223 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1737226 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1737227 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1737230 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1737230 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1737233 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1737234 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1737242 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1737243 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1737246 T4075 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1737246 T4075 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1737257 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1737259 T4086 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4244bc55 name:ZooKeeperConnection Watcher:127.0.0.1:48841/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1737259 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1738277 T4075 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1738281 T4075 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1738284 T4075 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:26574
   [junit4]   2> 1738286 T4075 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1738287 T4075 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1738287 T4075 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-002
   [junit4]   2> 1738287 T4075 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-002/'
   [junit4]   2> 1738314 T4075 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-002/solr.xml
   [junit4]   2> 1738382 T4075 oasc.CoreContainer.<init> New CoreContainer 13078740
   [junit4]   2> 1738383 T4075 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-002/]
   [junit4]   2> 1738384 T4075 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1738384 T4075 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1738384 T4075 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1738385 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1738385 T4075 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1738385 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1738385 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1738386 T4075 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1738386 T4075 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1738389 T4075 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1738389 T4075 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1738390 T4075 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1738390 T4075 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48841/solr
   [junit4]   2> 1738390 T4075 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1738391 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1738394 T4097 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16dd6a1c name:ZooKeeperConnection Watcher:127.0.0.1:48841 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1738394 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1738408 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1738409 T4099 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69907f86 name:ZooKeeperConnection Watcher:127.0.0.1:48841/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1738410 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1738412 T4075 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1738415 T4075 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1738427 T4075 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1738431 T4075 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1738442 T4075 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1738446 T4075 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1738453 T4075 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:26574_nm_hb%2Fel
   [junit4]   2> 1738454 T4075 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:26574_nm_hb%2Fel
   [junit4]   2> 1738457 T4075 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1738466 T4075 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1738477 T4075 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:26574_nm_hb%2Fel
   [junit4]   2> 1738477 T4075 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1738479 T4075 oasc.Overseer.start Overseer (id=91808340197900292-127.0.0.1:26574_nm_hb%2Fel-n_0000000000) starting
   [junit4]   2> 1738489 T4075 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1738506 T4101 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1738507 T4075 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1738510 T4075 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1738511 T4075 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1738514 T4100 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1738521 T4102 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1738521 T4102 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1738522 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1738522 T4102 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1738524 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1738524 T4100 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:26574/nm_hb/el",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:26574_nm_hb%2Fel",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1738524 T4100 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1738525 T4100 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1738527 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1738528 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1739524 T4102 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1739524 T4102 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1739525 T4102 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1739525 T4102 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1739526 T4102 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1739526 T4102 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-002/collection1/'
   [junit4]   2> 1739528 T4102 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1739529 T4102 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1739529 T4102 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1739603 T4102 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1739676 T4102 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1739777 T4102 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1739788 T4102 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1740317 T4102 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1740321 T4102 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1740323 T4102 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1740329 T4102 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1740352 T4102 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1740352 T4102 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1740413 T4102 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-002/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/control/data/
   [junit4]   2> 1740413 T4102 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f08589b
   [junit4]   2> 1740415 T4102 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/control/data
   [junit4]   2> 1740416 T4102 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/control/data/index/
   [junit4]   2> 1740417 T4102 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1740418 T4102 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/control/data/index
   [junit4]   2> 1740419 T4102 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=513088945, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1740424 T4102 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1740424 T4102 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1740429 T4102 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1740429 T4102 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1740429 T4102 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1740430 T4102 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1740430 T4102 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1740430 T4102 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1740431 T4102 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1740431 T4102 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1740431 T4102 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1740432 T4102 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1740432 T4102 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1740432 T4102 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1740433 T4102 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1740433 T4102 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1740434 T4102 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1740434 T4102 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1740434 T4102 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1740451 T4102 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1740455 T4102 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1740456 T4102 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1740456 T4102 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=50.1259765625, floorSegmentMB=0.556640625, forceMergeDeletesPctAllowed=19.016386616773776, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1740458 T4102 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1740458 T4102 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1740459 T4102 oass.SolrIndexSearcher.<init> Opening Searcher@706222ab[collection1] main
   [junit4]   2> 1740459 T4102 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1740460 T4102 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1740461 T4102 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1740461 T4102 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1740461 T4102 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1740462 T4102 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1740462 T4102 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1740463 T4102 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1740463 T4102 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1740463 T4102 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1740466 T4103 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@706222ab[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1740468 T4102 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1740468 T4106 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:26574/nm_hb/el collection:control_collection shard:shard1
   [junit4]   2> 1740469 T4075 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 1740469 T4075 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1740470 T4106 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1740475 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1740484 T4108 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fee3f61 name:ZooKeeperConnection Watcher:127.0.0.1:48841/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1740485 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1740487 T4075 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1740497 T4106 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1740504 T4075 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1740505 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1740506 T4106 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1740506 T4106 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C8246 name=collection1 org.apache.solr.core.SolrCore@5ec1a07d url=https://127.0.0.1:26574/nm_hb/el/collection1 node=127.0.0.1:26574_nm_hb%2Fel C8246_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:26574/nm_hb/el, core=collection1, node_name=127.0.0.1:26574_nm_hb%2Fel}
   [junit4]   2> 1740506 T4106 C8246 P26574 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:26574/nm_hb/el/collection1/
   [junit4]   2> 1740507 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1740507 T4106 C8246 P26574 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1740507 T4106 C8246 P26574 oasc.SyncStrategy.syncToMe https://127.0.0.1:26574/nm_hb/el/collection1/ has no replicas
   [junit4]   2> 1740507 T4106 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:26574/nm_hb/el/collection1/ shard1
   [junit4]   2> 1740508 T4106 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1740510 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1740520 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1740520 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1740541 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1740545 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1740649 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1740649 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1740694 T4106 oasc.ZkController.register We are https://127.0.0.1:26574/nm_hb/el/collection1/ and leader is https://127.0.0.1:26574/nm_hb/el/collection1/
   [junit4]   2> 1740694 T4106 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:26574/nm_hb/el
   [junit4]   2> 1740694 T4106 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1740694 T4106 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1740694 T4106 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1740696 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1740697 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1740697 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1740697 T4106 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1740698 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1740699 T4100 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:26574/nm_hb/el",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:26574_nm_hb%2Fel",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1740707 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1740810 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1740810 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1741577 T4075 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1741577 T4075 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1741581 T4075 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1741583 T4075 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:34672
   [junit4]   2> 1741585 T4075 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1741585 T4075 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1741585 T4075 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-003
   [junit4]   2> 1741586 T4075 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-003/'
   [junit4]   2> 1741613 T4075 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-003/solr.xml
   [junit4]   2> 1741681 T4075 oasc.CoreContainer.<init> New CoreContainer 1898324190
   [junit4]   2> 1741681 T4075 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-003/]
   [junit4]   2> 1741682 T4075 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1741683 T4075 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1741683 T4075 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1741683 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1741684 T4075 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1741684 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1741684 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1741684 T4075 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1741685 T4075 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1741688 T4075 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1741688 T4075 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1741688 T4075 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1741689 T4075 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48841/solr
   [junit4]   2> 1741689 T4075 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1741690 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1741693 T4119 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35a19702 name:ZooKeeperConnection Watcher:127.0.0.1:48841 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1741693 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1741711 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1741713 T4121 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55f66d5 name:ZooKeeperConnection Watcher:127.0.0.1:48841/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1741713 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1741729 T4075 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1742732 T4075 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34672_nm_hb%2Fel
   [junit4]   2> 1742736 T4075 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34672_nm_hb%2Fel
   [junit4]   2> 1742750 T4122 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1742750 T4122 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1742752 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1742752 T4122 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1742752 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1742753 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1742754 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1742755 T4100 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:34672/nm_hb/el",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:34672_nm_hb%2Fel",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1742755 T4100 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 1742755 T4100 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1742759 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1742760 T4121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1742760 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1742760 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1743754 T4122 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1743754 T4122 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1743755 T4122 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1743755 T4122 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1743756 T4122 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1743756 T4122 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-003/collection1/'
   [junit4]   2> 1743758 T4122 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1743759 T4122 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1743759 T4122 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1743834 T4122 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1743911 T4122 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1744013 T4122 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1744024 T4122 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1744555 T4122 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1744559 T4122 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1744561 T4122 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1744567 T4122 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1744590 T4122 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1744590 T4122 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1744591 T4122 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-003/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty1/
   [junit4]   2> 1744591 T4122 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f08589b
   [junit4]   2> 1744592 T4122 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty1
   [junit4]   2> 1744594 T4122 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty1/index/
   [junit4]   2> 1744595 T4122 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1744596 T4122 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty1/index
   [junit4]   2> 1744597 T4122 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=513088945, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1744601 T4122 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1744602 T4122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1744607 T4122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1744608 T4122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1744608 T4122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1744608 T4122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1744609 T4122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1744609 T4122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1744610 T4122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1744610 T4122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1744610 T4122 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1744611 T4122 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1744611 T4122 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1744612 T4122 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1744612 T4122 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1744613 T4122 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1744613 T4122 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1744614 T4122 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1744614 T4122 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1744633 T4122 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1744637 T4122 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1744637 T4122 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1744638 T4122 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=50.1259765625, floorSegmentMB=0.556640625, forceMergeDeletesPctAllowed=19.016386616773776, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1744640 T4122 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 1744640 T4122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1744641 T4122 oass.SolrIndexSearcher.<init> Opening Searcher@3607e8ca[collection1] main
   [junit4]   2> 1744641 T4122 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1744642 T4122 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1744643 T4122 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1744643 T4122 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1744643 T4122 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1744644 T4122 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1744644 T4122 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1744645 T4122 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1744645 T4122 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1744645 T4122 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1744650 T4123 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3607e8ca[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1744653 T4122 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1744654 T4126 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:34672/nm_hb/el collection:collection1 shard:shard2
   [junit4]   2> 1744654 T4075 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 1744655 T4075 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1744655 T4126 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1744675 T4126 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1744687 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1744687 T4126 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1744688 T4126 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C8247 name=collection1 org.apache.solr.core.SolrCore@335ab1be url=https://127.0.0.1:34672/nm_hb/el/collection1 node=127.0.0.1:34672_nm_hb%2Fel C8247_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:34672/nm_hb/el, core=collection1, node_name=127.0.0.1:34672_nm_hb%2Fel}
   [junit4]   2> 1744688 T4126 C8247 P34672 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:34672/nm_hb/el/collection1/
   [junit4]   2> 1744688 T4126 C8247 P34672 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1744688 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1744688 T4126 C8247 P34672 oasc.SyncStrategy.syncToMe https://127.0.0.1:34672/nm_hb/el/collection1/ has no replicas
   [junit4]   2> 1744689 T4126 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:34672/nm_hb/el/collection1/ shard2
   [junit4]   2> 1744689 T4126 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1744698 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1744706 T4121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1744706 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1744706 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1744720 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1744723 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1744826 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1744826 T4121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1744826 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1744873 T4126 oasc.ZkController.register We are https://127.0.0.1:34672/nm_hb/el/collection1/ and leader is https://127.0.0.1:34672/nm_hb/el/collection1/
   [junit4]   2> 1744873 T4126 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:34672/nm_hb/el
   [junit4]   2> 1744873 T4126 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1744873 T4126 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1744873 T4126 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1744875 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1744876 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1744876 T4126 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1744876 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1744877 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1744878 T4100 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:34672/nm_hb/el",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:34672_nm_hb%2Fel",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 1744881 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1744984 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1744984 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1744984 T4121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1745724 T4075 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1745725 T4075 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1745728 T4075 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1745730 T4075 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:63340
   [junit4]   2> 1745732 T4075 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1745732 T4075 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1745732 T4075 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-004
   [junit4]   2> 1745733 T4075 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-004/'
   [junit4]   2> 1745760 T4075 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-004/solr.xml
   [junit4]   2> 1745829 T4075 oasc.CoreContainer.<init> New CoreContainer 690217273
   [junit4]   2> 1745830 T4075 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-004/]
   [junit4]   2> 1745831 T4075 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1745831 T4075 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1745832 T4075 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1745832 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1745832 T4075 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1745832 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1745833 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1745833 T4075 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1745833 T4075 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1745836 T4075 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1745837 T4075 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1745837 T4075 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1745837 T4075 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48841/solr
   [junit4]   2> 1745838 T4075 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1745838 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1745842 T4137 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@600c284f name:ZooKeeperConnection Watcher:127.0.0.1:48841 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1745842 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1745856 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1745858 T4139 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@79abbc0a name:ZooKeeperConnection Watcher:127.0.0.1:48841/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1745858 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1745868 T4075 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1746871 T4075 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63340_nm_hb%2Fel
   [junit4]   2> 1746874 T4075 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63340_nm_hb%2Fel
   [junit4]   2> 1746892 T4140 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1746893 T4140 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1746895 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1746895 T4140 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1746895 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1746895 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1746896 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1746897 T4100 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63340/nm_hb/el",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63340_nm_hb%2Fel",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1746897 T4100 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1746897 T4100 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1746901 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1746908 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1746908 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1746908 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1746908 T4121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1747896 T4140 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1747896 T4140 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1747897 T4140 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1747897 T4140 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1747898 T4140 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1747898 T4140 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-004/collection1/'
   [junit4]   2> 1747900 T4140 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 1747901 T4140 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 1747902 T4140 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 1747985 T4140 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 1748058 T4140 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1748160 T4140 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1748171 T4140 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1748702 T4140 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1748706 T4140 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1748708 T4140 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1748714 T4140 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1748737 T4140 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1748738 T4140 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 1748738 T4140 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-004/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty2/
   [junit4]   2> 1748738 T4140 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6f08589b
   [junit4]   2> 1748740 T4140 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty2
   [junit4]   2> 1748741 T4140 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty2/index/
   [junit4]   2> 1748742 T4140 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1748743 T4140 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty2/index
   [junit4]   2> 1748744 T4140 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=513088945, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1748752 T4140 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1748752 T4140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1748758 T4140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1748758 T4140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1748759 T4140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1748759 T4140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1748759 T4140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1748759 T4140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1748760 T4140 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1748760 T4140 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1748761 T4140 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1748761 T4140 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1748762 T4140 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1748762 T4140 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1748762 T4140 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1748763 T4140 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1748764 T4140 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1748764 T4140 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1748765 T4140 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1748783 T4140 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1748788 T4140 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1748788 T4140 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1748789 T4140 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=40, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=50.1259765625, floorSegmentMB=0.556640625, forceMergeDeletesPctAllowed=19.016386616773776, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1748791 T4140 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 1748791 T4140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1748791 T4140 oass.SolrIndexSearcher.<init> Opening Searcher@64e2c9bc[collection1] main
   [junit4]   2> 1748792 T4140 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1748793 T4140 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1748793 T4140 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1748794 T4140 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1748794 T4140 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1748794 T4140 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1748795 T4140 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1748795 T4140 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1748796 T4140 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 1748796 T4140 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1748800 T4141 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@64e2c9bc[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 1748803 T4140 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1748804 T4144 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:63340/nm_hb/el collection:collection1 shard:shard1
   [junit4]   2> 1748804 T4075 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 1748805 T4075 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1748805 T4144 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1748820 T4144 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1748831 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1748831 T4144 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1748832 T4144 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C8248 name=collection1 org.apache.solr.core.SolrCore@21839b9 url=https://127.0.0.1:63340/nm_hb/el/collection1 node=127.0.0.1:63340_nm_hb%2Fel C8248_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:63340/nm_hb/el, core=collection1, node_name=127.0.0.1:63340_nm_hb%2Fel}
   [junit4]   2> 1748832 T4144 C8248 P63340 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:63340/nm_hb/el/collection1/
   [junit4]   2> 1748832 T4144 C8248 P63340 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1748833 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1748833 T4144 C8248 P63340 oasc.SyncStrategy.syncToMe https://127.0.0.1:63340/nm_hb/el/collection1/ has no replicas
   [junit4]   2> 1748833 T4144 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:63340/nm_hb/el/collection1/ shard1
   [junit4]   2> 1748833 T4144 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1748836 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1748837 T4121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1748837 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1748837 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1748837 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1748841 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1748845 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1748969 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1748969 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1748969 T4121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1748969 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1749045 T4144 oasc.ZkController.register We are https://127.0.0.1:63340/nm_hb/el/collection1/ and leader is https://127.0.0.1:63340/nm_hb/el/collection1/
   [junit4]   2> 1749045 T4144 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:63340/nm_hb/el
   [junit4]   2> 1749045 T4144 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1749045 T4144 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1749046 T4144 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1749048 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1749048 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1749048 T4144 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1749048 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1749049 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1749050 T4100 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63340/nm_hb/el",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63340_nm_hb%2Fel",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 1749079 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1749182 T4099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1749183 T4108 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1749183 T4139 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1749183 T4121 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1749996 T4075 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1749997 T4075 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1750001 T4075 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1750002 T4075 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:59538
   [junit4]   2> 1750004 T4075 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1750005 T4075 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1750005 T4075 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-005
   [junit4]   2> 1750005 T4075 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-005/'
   [junit4]   2> 1750032 T4075 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-005/solr.xml
   [junit4]   2> 1750100 T4075 oasc.CoreContainer.<init> New CoreContainer 1344864280
   [junit4]   2> 1750101 T4075 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-005/]
   [junit4]   2> 1750102 T4075 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1750102 T4075 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1750102 T4075 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1750103 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1750103 T4075 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1750103 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1750104 T4075 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1750104 T4075 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1750104 T4075 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1750107 T4075 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1750107 T4075 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1750108 T4075 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1750108 T4075 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:48841/solr
   [junit4]   2> 1750108 T4075 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1750109 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1750112 T4155 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13301edb name:ZooKeeperConnection Watcher:127.0.0.1:48841 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1750113 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1750116 T4075 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1750117 T4157 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@293006db name:ZooKeeperConnection Watcher:127.0.0.1:48841/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1750117 T4075 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1750138 T4075 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1751141 T4075 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59538_nm_hb%2Fel
   [junit4]   2> 1751144 T4075 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59538_nm_hb%2Fel
   [junit4]   2> 1751159 T4158 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1751159 T4158 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1751161 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1751161 T4158 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1751161 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1751162 T4099 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1751163 T4100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1751164 T4100 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:59538/nm_hb/el",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:59538_nm_hb%2Fel",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 1751164 T4100 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 1751164 T4100 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 1751167 T4099 oasc.DistributedQueue$LatchChildWatcher.pro

[...truncated too long message...]

ir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty11/index;done=false>>]
   [junit4]   2> 2042972 T4221 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2042971 T4296 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty9
   [junit4]   2> 2042972 T4346 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty11/index
   [junit4]   2> 2042973 T4296 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty9/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty9/index;done=false>>]
   [junit4]   2> 2042973 T4346 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty11 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty11;done=false>>]
   [junit4]   2> 2042973 T4296 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty9/index
   [junit4]   2> 2042973 T4346 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001/tempDir-001/jetty11
   [junit4]   2> 2042974 T4296 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2042974 T4346 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 10)
   [junit4]   2> 2042974 T4296 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 2042974 T4346 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2042975 T4346 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2042976 T4346 oasc.LeaderElector$ElectionWatcher.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-51907E065F3FC472-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene46), sim=DefaultSimilarity, locale=ar_LB, timezone=America/Creston
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=138057968,total=500695040
   [junit4]   2> NOTE: All tests run in this JVM: [SolrPluginUtilsTest, SSLMigrationTest, TestDocBasedVersionConstraints, CurrencyFieldXmlFileTest, QueryFacetTest, SuggesterWFSTTest, TestAddFieldRealTimeGet, TestHashPartitioner, TestIndexSearcher, SuggestComponentTest, SolrRequestParserTest, XsltUpdateRequestHandlerTest, TestModifyConfFiles, HdfsSyncSliceTest, RemoteQueryErrorTest, TestStressRecovery, TestPartialUpdateDeduplication, TestNoOpRegenerator, FileUtilsTest, TestManagedSchema, RequestHandlersTest, HardAutoCommitTest, TestRTGBase, TestReload, TestStressUserVersions, TriLevelCompositeIdRoutingTest, DeleteInactiveReplicaTest, ResourceLoaderTest, CursorMarkTest, DOMUtilTest, AssignTest, TestSolrXMLSerializer, TestStressReorder, TestLMDirichletSimilarityFactory, BlockCacheTest, TestQuerySenderListener, DistributedDebugComponentTest, TestPhraseSuggestions, SolrTestCaseJ4Test, TestSchemaSimilarityResource, TestManagedResource, SuggesterTSTTest, ConvertedLegacyTest, SystemInfoHandlerTest, TestFieldTypeResource, UniqFieldsUpdateProcessorFactoryTest, DisMaxRequestHandlerTest, TestJmxIntegration, TestOmitPositions, ShardRoutingCustomTest, MultiTermTest, TestSerializedLuceneMatchVersion, TestCoreDiscovery, TestSearcherReuse, ConnectionManagerTest, QueryResultKeyTest, TestComponentsName, TestRandomFaceting, LegacyHTMLStripCharFilterTest, TestWordDelimiterFilterFactory, TestXIncludeConfig, CoreAdminRequestStatusTest, DocValuesMissingTest, BasicZkTest, TestBinaryResponseWriter, OpenExchangeRatesOrgProviderTest, SortByFunctionTest, TestQuerySenderNoQuery, BadCopyFieldTest, AlternateDirectoryTest, OpenCloseCoreStressTest, AnalysisErrorHandlingTest, TestOverriddenPrefixQueryForCustomFieldType, TestFastWriter, PluginInfoTest, DirectSolrConnectionTest, TestCollectionAPI, SignatureUpdateProcessorFactoryTest, TestSweetSpotSimilarityFactory, HighlighterConfigTest, CachingDirectoryFactoryTest, StandardRequestHandlerTest, TestFuzzyAnalyzedSuggestions, TestSolrQueryParserResource, ChangedSchemaMergeTest, TestDynamicFieldResource, SimplePostToolTest, RangeFacetTest, HdfsCollectionsAPIDistributedZkTest, HdfsLockFactoryTest, SolrIndexConfigTest, FieldFacetExtrasTest, TestSolrJ, TestCloudInspectUtil, ChaosMonkeySafeLeaderTest]
   [junit4] Completed on J1 in 306.86s, 1 test, 1 error <<< FAILURES!

[...truncated 754 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1296: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:920: There were test failures: 406 suites, 1688 tests, 1 error, 34 ignored (5 assumptions)

Total time: 212 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 561 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/561/

5 tests failed.
REGRESSION:  org.apache.lucene.index.TestBagOfPositions.test

Error Message:
Captured an uncaught exception in thread: Thread[id=6634, name=Thread-5938, state=RUNNABLE, group=TGRP-TestBagOfPositions]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6634, name=Thread-5938, state=RUNNABLE, group=TGRP-TestBagOfPositions]
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
	at org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:123)
Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:113)
	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
	at org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:120)


REGRESSION:  org.apache.lucene.index.TestIndexWriterDelete.testDeleteAllNoDeadLock

Error Message:
Captured an uncaught exception in thread: Thread[id=4425, name=Thread-4027, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4425, name=Thread-4027, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:108)
	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterDelete

Error Message:
The test or suite printed 20288 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true

Stack Trace:
java.lang.AssertionError: The test or suite printed 20288 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
	at org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:210)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
	at java.lang.Thread.run(Thread.java:724)


REGRESSION:  org.apache.lucene.index.TestIndexWriterExceptions2.testBasics

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterExceptions2

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)




Build Log:
[...truncated 756 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterDelete
   [junit4]   2> May 09, 2014 7:28:42 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-4026,5,TGRP-TestIndexWriterDelete]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.maybeCommit(RandomIndexWriter.java:155)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:151)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)
   [junit4]   2> 
   [junit4]   2> May 09, 2014 7:28:42 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-4028,5,TGRP-TestIndexWriterDelete]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.maybeCommit(RandomIndexWriter.java:155)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:151)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)
   [junit4]   2> 
   [junit4]   2> May 09, 2014 7:28:42 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-4023,5,TGRP-TestIndexWriterDelete]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.maybeCommit(RandomIndexWriter.java:155)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:151)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)
   [junit4]   2> 
   [junit4]   2> May 09, 2014 7:28:42 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-4027,5,TGRP-TestIndexWriterDelete]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:108)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)
   [junit4]   2> 
   [junit4]   2> May 09, 2014 7:28:42 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-4029,5,TGRP-TestIndexWriterDelete]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.maybeCommit(RandomIndexWriter.java:155)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:151)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)
   [junit4]   2> 
   [junit4]   2> May 09, 2014 7:28:42 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-4025,5,TGRP-TestIndexWriterDelete]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:113)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)
   [junit4]   2> 
   [junit4]   2> May 09, 2014 7:28:42 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-4024,5,TGRP-TestIndexWriterDelete]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:108)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)
   [junit4]   2> 
   [junit4]   2> May 09, 2014 7:28:42 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-4032,5,TGRP-TestIndexWriterDelete]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:113)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)
   [junit4]   2> 
   [junit4]   2> May 09, 2014 7:28:42 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-4030,5,TGRP-TestIndexWriterDelete]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:108)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterDelete -Dtests.method=testDeleteAllNoDeadLock -Dtests.seed=AE6C3EFA0CEB6D9E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=en_CA -Dtests.timezone=Etc/GMT-4 -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.32s J0 | TestIndexWriterDelete.testDeleteAllNoDeadLock <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4425, name=Thread-4027, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:108)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4424, name=Thread-4026, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.maybeCommit(RandomIndexWriter.java:155)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:151)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4421, name=Thread-4023, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.maybeCommit(RandomIndexWriter.java:155)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:151)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4426, name=Thread-4028, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.maybeCommit(RandomIndexWriter.java:155)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:151)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4427, name=Thread-4029, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.maybeCommit(RandomIndexWriter.java:155)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:151)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4423, name=Thread-4025, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:113)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4422, name=Thread-4024, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:108)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)Throwable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4430, name=Thread-4032, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:113)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)Throwable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4428, name=Thread-4030, state=RUNNABLE, group=TGRP-TestIndexWriterDelete]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:339)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:108)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterDelete$1.run(TestIndexWriterDelete.java:333)
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=207), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=207)), sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {field=DFR I(F)1, contents=DFR GB2, id=DFR G2, content=IB SPL-L2, body=IB SPL-D2, city=IB LL-D2}, locale=en_CA, timezone=Etc/GMT-4
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=238625968,total=427360256
   [junit4]   2> NOTE: All tests run in this JVM: [Nested1, TestMultiPhraseQuery, TestBytesRefHash, TestPagedBytes, TestDeterminism, TestScorerPerf, TestSortRandom, TestSimpleExplanationsOfNonMatches, TestPayloads, TestBooleanMinShouldMatch, TestDisjunctionMaxQuery, TestSegmentReader, TestSetOnce, TestLucene40PostingsReader, TestTopDocsMerge, TestCompressingTermVectorsFormat, TestMatchAllDocsQuery, Test4GBStoredFields, TestField, TestTermsEnum2, TestOfflineSorter, TestTermRangeFilter, TestNRTReaderWithThreads, TestAssertions, TestTopFieldCollector, TestNumericRangeQuery64, TestFastCompressionMode, TestRateLimitedDirectoryWrapper, TestBinaryDocValuesUpdates, TestCustomSearcherSort, TestFieldCacheRewriteMethod, TestRollback, TestDeterminizeLexicon, TestDemo, TestSizeBoundedForceMerge, TestUniqueTermCount, TestCachingTokenFilter, TestExternalCodecs, TestBasicOperations, TestCharTermAttributeImpl, TestNumericDocValuesUpdates, TestIndexWriterOutOfMemory, TestReuseDocsEnum, TestMultiTermConstantScore, TestSloppyPhraseQuery2, TestNumericRangeQuery32, TestByteSlices, TestPhraseQuery, TestPrefixRandom, TestComplexExplanations, TestBinaryDocument, TestSloppyMath, TestRamUsageEstimator, TestBroadWord, TestParallelReaderEmptyIndex, TestQueryWrapperFilter, TestPostingsOffsets, Test2BNumericDocValues, TestMergeSchedulerExternal, TestSort, TestRecyclingIntBlockAllocator, TestLucene40PostingsFormat, TestDirectoryReader, TestAutomatonQueryUnicode, TestPayloadTermQuery, TestFieldsReader, TestTermsEnum, Test2BPostings, TestDocBoost, TestFlushByRamOrCountsPolicy, TestWAH8DocIdSet, FuzzyTermOnShortTermsTest, TestCompiledAutomaton, TestParallelTermEnum, TestUTF32ToUTF8, TestIndexCommit, TestQueryRescorer, Nested1, TestFixedBitSet, TestStressIndexing2, TestDocTermOrdsRewriteMethod, TestEarlyTermination, TestControlledRealTimeReopenThread, TestDeletionPolicy, TestBytesRef, TestIndexWriterCommit, TestCustomNorms, Nested, Nested, TestTransactionRollback, TestLookaheadTokenFilter, TestCompoundFile, TestBooleanScorer, TestTermVectorsWriter, TestIndexWriterOutOfFileDescriptors, TestStressAdvance, TestPostingsFormat, TestSpansAdvanced2, TestDocumentsWriterStallControl, TestIntroSorter, TestCharsRef, TestIndexWriter, TestIndexWriterReader, TestTermVectorsFormat, TestIndexWriterMerging, TestFSTs, TestBytesStore, TestPackedInts, TestLucene42DocValuesFormat, TestBackwardsCompatibility, TestShardSearching, TestBlockPostingsFormat3, TestMultiMMap, TestSearcherManager, TestBooleanOr, TestBasics, TestSloppyPhraseQuery, TestIndexWriterDelete]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterDelete -Dtests.seed=AE6C3EFA0CEB6D9E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=en_CA -Dtests.timezone=Etc/GMT-4 -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J0 | TestIndexWriterDelete (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: The test or suite printed 20288 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:724)
   [junit4] Completed on J0 in 26.40s, 22 tests, 1 failure, 1 error <<< FAILURES!

[...truncated 155 lines...]
   [junit4] Suite: org.apache.lucene.index.TestBagOfPositions
   [junit4]   2> svi 09, 2014 6:42:56 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-5938,5,TGRP-TestBagOfPositions]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:123)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:113)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]   2> 	at org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:120)
   [junit4]   2> 
   [junit4]   2> svi 09, 2014 6:43:00 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-5937,5,TGRP-TestBagOfPositions]
   [junit4]   2> java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:123)
   [junit4]   2> Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]   2> 	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
   [junit4]   2> 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:113)
   [junit4]   2> 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]   2> 	at org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:120)
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBagOfPositions -Dtests.method=test -Dtests.seed=AE6C3EFA0CEB6D9E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=hr -Dtests.timezone=Europe/Zaporozhye -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   21.4s J0 | TestBagOfPositions.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: expected:<157> but was:<156>
   [junit4]    > 	at org.apache.lucene.index.TestBagOfPositions.test(TestBagOfPositions.java:145)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:724)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6634, name=Thread-5938, state=RUNNABLE, group=TGRP-TestBagOfPositions]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:123)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setMaxBufferedDocs(LiveIndexWriterConfig.java:298)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1046)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:113)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]    > 	at org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:120)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6633, name=Thread-5937, state=RUNNABLE, group=TGRP-TestBagOfPositions]
   [junit4]    > Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:123)
   [junit4]    > Caused by: java.lang.IllegalArgumentException: at least one of ramBufferSize and maxBufferedDocs must be enabled
   [junit4]    > 	at org.apache.lucene.index.LiveIndexWriterConfig.setRAMBufferSizeMB(LiveIndexWriterConfig.java:257)
   [junit4]    > 	at org.apache.lucene.util.LuceneTestCase.maybeChangeLiveIndexWriterConfig(LuceneTestCase.java:1055)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:113)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:109)
   [junit4]    > 	at org.apache.lucene.index.TestBagOfPositions$1.run(TestBagOfPositions.java:120)
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/core/test/J0/./lucene.index.TestBagOfPositions-AE6C3EFA0CEB6D9E-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=hr, timezone=Europe/Zaporozhye
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=284153688,total=467599360
   [junit4]   2> NOTE: All tests run in this JVM: [Nested1, TestMultiPhraseQuery, TestBytesRefHash, TestPagedBytes, TestDeterminism, TestScorerPerf, TestSortRandom, TestSimpleExplanationsOfNonMatches, TestPayloads, TestBooleanMinShouldMatch, TestDisjunctionMaxQuery, TestSegmentReader, TestSetOnce, TestLucene40PostingsReader, TestTopDocsMerge, TestCompressingTermVectorsFormat, TestMatchAllDocsQuery, Test4GBStoredFields, TestField, TestTermsEnum2, TestOfflineSorter, TestTermRangeFilter, TestNRTReaderWithThreads, TestAssertions, TestTopFieldCollector, TestNumericRangeQuery64, TestFastCompressionMode, TestRateLimitedDirectoryWrapper, TestBinaryDocValuesUpdates, TestCustomSearcherSort, TestFieldCacheRewriteMethod, TestRollback, TestDeterminizeLexicon, TestDemo, TestSizeBoundedForceMerge, TestUniqueTermCount, TestCachingTokenFilter, TestExternalCodecs, TestBasicOperations, TestCharTermAttributeImpl, TestNumericDocValuesUpdates, TestIndexWriterOutOfMemory, TestReuseDocsEnum, TestMultiTermConstantScore, TestSloppyPhraseQuery2, TestNumericRangeQuery32, TestByteSlices, TestPhraseQuery, TestPrefixRandom, TestComplexExplanations, TestBinaryDocument, TestSloppyMath, TestRamUsageEstimator, TestBroadWord, TestParallelReaderEmptyIndex, TestQueryWrapperFilter, TestPostingsOffsets, Test2BNumericDocValues, TestMergeSchedulerExternal, TestSort, TestRecyclingIntBlockAllocator, TestLucene40PostingsFormat, TestDirectoryReader, TestAutomatonQueryUnicode, TestPayloadTermQuery, TestFieldsReader, TestTermsEnum, Test2BPostings, TestDocBoost, TestFlushByRamOrCountsPolicy, TestWAH8DocIdSet, FuzzyTermOnShortTermsTest, TestCompiledAutomaton, TestParallelTermEnum, TestUTF32ToUTF8, TestIndexCommit, TestQueryRescorer, Nested1, TestFixedBitSet, TestStressIndexing2, TestDocTermOrdsRewriteMethod, TestEarlyTermination, TestControlledRealTimeReopenThread, TestDeletionPolicy, TestBytesRef, TestIndexWriterCommit, TestCustomNorms, Nested, Nested, TestTransactionRollback, TestLookaheadTokenFilter, TestCompoundFile, TestBooleanScorer, TestTermVectorsWriter, TestIndexWriterOutOfFileDescriptors, TestStressAdvance, TestPostingsFormat, TestSpansAdvanced2, TestDocumentsWriterStallControl, TestIntroSorter, TestCharsRef, TestIndexWriter, TestIndexWriterReader, TestTermVectorsFormat, TestIndexWriterMerging, TestFSTs, TestBytesStore, TestPackedInts, TestLucene42DocValuesFormat, TestBackwardsCompatibility, TestShardSearching, TestBlockPostingsFormat3, TestMultiMMap, TestSearcherManager, TestBooleanOr, TestBasics, TestSloppyPhraseQuery, TestIndexWriterDelete, TestLucene40DocValuesFormat, TestIndexWriterMergePolicy, TestNRTThreads, TestDuelingCodecs, TestPersistentSnapshotDeletionPolicy, TestAtomicUpdate, TestTimeLimitingCollector, TestSearchWithThreads, TestSearchAfter, TestConcurrentMergeScheduler, TestHighCompressionMode, TestStressIndexing, TestMinimize, TestRollingUpdates, TestBlockPostingsFormat2, TestSnapshotDeletionPolicy, TestTieredMergePolicy, TestIndexWriterForceMerge, TestArrayUtil, TestBufferedIndexInput, TestConsistentFieldNumbers, TestSimpleExplanations, TestDocTermOrdsRangeFilter, TestLockFactory, TestCollectionUtil, TestBagOfPositions]
   [junit4] Completed on J0 in 21.45s, 1 test, 1 error <<< FAILURES!

[...truncated 510 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterExceptions2
   [junit4]   2> may 09, 2014 12:05:19 AM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.TestIndexWriterExceptions2
   [junit4]   2> ==== jstack at approximately timeout time ====
   [junit4]   2> "TEST-TestIndexWriterExceptions2.testBasics-seed#[AE6C3EFA0CEB6D9E]" ID=87 TIMED_WAITING on org.apache.lucene.index.IndexWriter@3578152e
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- timed waiting on org.apache.lucene.index.IndexWriter@3578152e
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4200)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2237)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.finishMerges(IndexWriter.java:2221)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:885)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:869)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterExceptions2.testBasics(TestIndexWriterExceptions2.java:198)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> 
   [junit4]   2> "SUITE-TestIndexWriterExceptions2-seed#[AE6C3EFA0CEB6D9E]" ID=86 RUNNABLE
   [junit4]   2> 	at sun.management.ThreadImpl.dumpThreads0(Native Method)
   [junit4]   2> 	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:666)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$900(ThreadLeakControl.java:62)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:407)
   [junit4]   2> 	- locked java.lang.Object@5eb69a6e
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:642)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:129)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:559)
   [junit4]   2> 
   [junit4]   2> "Signal Dispatcher" ID=4 RUNNABLE
   [junit4]   2> 
   [junit4]   2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@2654dc56
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@2654dc56
   [junit4]   2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
   [junit4]   2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
   [junit4]   2> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
   [junit4]   2> 
   [junit4]   2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@d4da5c3
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on java.lang.ref.Reference$Lock@d4da5c3
   [junit4]   2> 	at java.lang.Object.wait(Object.java:503)
   [junit4]   2> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
   [junit4]   2> 
   [junit4]   2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$1@2c23db2
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$1@2c23db2
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1260)
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1334)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:569)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:435)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:180)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:276)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
   [junit4]   2> 
   [junit4]   2> ^^==============================================
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterExceptions2 -Dtests.method=testBasics -Dtests.seed=AE6C3EFA0CEB6D9E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_ES -Dtests.timezone=Antarctica/Palmer -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   7200s J1 | TestIndexWriterExceptions2.testBasics <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4]   2> may 09, 2014 12:05:19 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> may 09, 2014 12:05:39 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.index.TestIndexWriterExceptions2: 
   [junit4]   2>    1) Thread[id=87, name=TEST-TestIndexWriterExceptions2.testBasics-seed#[AE6C3EFA0CEB6D9E], state=TIMED_WAITING, group=TGRP-TestIndexWriterExceptions2]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4200)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.waitForMerges(IndexWriter.java:2237)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.finishMerges(IndexWriter.java:2221)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:885)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:869)
   [junit4]   2>         at org.apache.lucene.index.TestIndexWriterExceptions2.testBasics(TestIndexWriterExceptions2.java:198)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   2>         at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:724)
   [junit4]   2> may 09, 2014 12:05:39 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=87, name=TEST-TestIndexWriterExceptions2.testBasics-seed#[AE6C3EFA0CEB6D9E], state=TIMED_WAITING, group=TGRP-TestIndexWriterExceptions2]
   [junit4]   1> Unexpected exception: dumping fake-exception-log:...
   [junit4]   1> 
   [junit4]   1> TEST: got expected fake exc:Fake IOException from TokenStream.end()
   [junit4]   1> java.io.IOException: Fake IOException from TokenStream.end()
   [junit4]   1> 	at org.apache.lucene.analysis.CrankyTokenFilter.end(CrankyTokenFilter.java:51)
   [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:651)
   [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:305)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:222)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:462)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1434)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1104)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1085)
   [junit4]   1> 	at org.apache.lucene.index.TestIndexWriterExceptions2.testBasics(TestIndexWriterExceptions2.java:126)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   1> 
   [junit4]   1> TEST: got expected fake exc:Fake IOException from TokenStream.incrementToken()
   [junit4]   1> java.io.IOException: Fake IOException from TokenStream.incrementToken()
   [junit4]   1> 	at org.apache.lucene.analysis.CrankyTokenFilter.incrementToken(CrankyTokenFilter.java:42)
   [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:587)
   [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:305)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:222)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:462)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1434)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1104)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1085)
   [junit4]   1> 	at org.apache.lucene.index.TestIndexWriterExceptions2.testBasics(TestIndexWriterExceptions2.java:126)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   1> 
   [junit4]   1> TEST: got expected fake exc:Fake IOException from TokenStream.incrementToken()
   [junit4]   1> java.io.IOException: Fake IOException from TokenStream.incrementToken()
   [junit4]   1> 	at org.apache.lucene.analysis.CrankyTokenFilter.incrementToken(CrankyTokenFilter.java:42)
   [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:587)
   [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:305)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:222)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:462)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1434)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1104)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1085)
   [junit4]   1> 	at org.apache.lucene.index.TestIndexWriterExceptions2.testBasics(TestIndexWriterExceptions2.java:126)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   1> 
   [junit4]   1> TEST: got expected fake exc:Fake IOException from TokenStream.incrementToken()
   [junit4]   1> java.io.IOException: Fake IOException from TokenStream.incrementToken()
   [junit4]   1> 	at org.apache.lucene.analysis.CrankyTokenFilter.incrementToken(CrankyTokenFilter.java:42)
   [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:587)
   [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:305)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:222)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:462)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1434)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1104)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1085)
   [junit4]   1> 	at org.apache.lucene.index.TestIndexWriterExceptions2.testBasics(TestIndexWriterExceptions2.java:126)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   1> 
   [junit4]   1> TEST: got expected fake exc:Fake IOException from TokenStream.reset()
   [junit4]   1> java.io.IOException: Fake IOException from TokenStream.reset()
   [junit4]   1> 	at org.apache.lucene.analysis.CrankyTokenFilter.reset(CrankyTokenFilter.java:60)
   [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:582)
   [junit4]   1> 	at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:305)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:222)
   [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:462)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1434)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1104)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1085)
   [junit4]   1> 	at org.apache.lucene.index.TestIndexWriterExceptions2.testBasics(TestIndexWriterExceptions2.java:126)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   1> 
   [junit4]   1> TEST: got expected fake exc:Fake IOException from FieldInfosFormat.getFieldInfosWriter()
   [junit4]   1> java.io.IOException: Fake IOException from FieldInfosFormat.getFieldInfosWriter()
   [junit4]   1> 	at org.apache.lucene.codecs.cranky.CrankyFieldInfosFormat.getFieldInfosWriter(CrankyFieldInfosFormat.java:47)
   [junit4]   1> 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:151)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3980)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3576)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1883)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:410)
   [junit4]   1> 	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:87)
   [junit4]   1> 	at org.apache.lucene.index.TestIndexWriterExceptions2.testBasics(TestIndexWriterExceptions2.java:175)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   1> 
   [junit4]   1> TEST: got expected fake exc:Fake IOException from StoredFieldsFormat.fieldsWriter()
   [junit4]   1> java.io.IOExceptio

[...truncated too long message...]

e(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   1> 
   [junit4]   1> TEST: got expected fake exc:Fake IOException from TermVectorsWriter.startField()
   [junit4]   1> java.io.IOException: Fake IOException from TermVectorsWriter.startField()
   [junit4]   1> 	at org.apache.lucene.codecs.cranky.CrankyTermVectorsFormat$CrankyTermVectorsWriter.startField(CrankyTermVectorsFormat.java:119)
   [junit4]   1> 	at org.apache.lucene.codecs.TermVectorsWriter.addAllDocVectors(TermVectorsWriter.java:257)
   [junit4]   1> 	at org.apache.lucene.codecs.TermVectorsWriter.merge(TermVectorsWriter.java:194)
   [junit4]   1> 	at org.apache.lucene.codecs.cranky.CrankyTermVectorsFormat$CrankyTermVectorsWriter.merge(CrankyTermVectorsFormat.java:79)
   [junit4]   1> 	at org.apache.lucene.index.SegmentMerger.mergeVectors(SegmentMerger.java:334)
   [junit4]   1> 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:142)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3980)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3576)
   [junit4]   1> 	at org.apache.lucene.index.SerialMergeScheduler.merge(SerialMergeScheduler.java:40)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1883)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2831)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2940)
   [junit4]   1> 	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2907)
   [junit4]   1> 	at org.apache.lucene.index.TestIndexWriterExceptions2.testBasics(TestIndexWriterExceptions2.java:181)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   1> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   [junit4]   1> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   1> 	at java.lang.reflect.Method.invoke(Method.java:606)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
   [junit4]   1> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
   [junit4]   1> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
   [junit4]   1> 	at java.lang.Thread.run(Thread.java:724)
   [junit4]   1> 
   [junit4]   2> May 09, 2014 4:05:40 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=497), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=497)), sim=DefaultSimilarity, locale=es_ES, timezone=Antarctica/Palmer
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=16,threads=1,free=193933544,total=290390016
   [junit4]   2> NOTE: All tests run in this JVM: [TestBooleanQuery, TestWeakIdentityMap, TestFieldCache, TestOmitPositions, TestLongPostings, TestSimpleAttributeImpl, TestPrefixFilter, TestGroupFiltering, TestStringHelper, TestSpanMultiTermQueryWrapper, Test2BDocs, TestMergedIterator, TestTopDocsCollector, TestTerm, TestFieldMaskingSpanQuery, Nested1, TestIndexWriterExceptions2]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterExceptions2 -Dtests.seed=AE6C3EFA0CEB6D9E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_ES -Dtests.timezone=Antarctica/Palmer -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | TestIndexWriterExceptions2 (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE6C3EFA0CEB6D9E]:0)
   [junit4] Completed on J1 in 7221.58s, 1 test, 2 errors <<< FAILURES!

[...truncated 264 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:474: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:447: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:49: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1296: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:920: There were test failures: 396 suites, 3061 tests, 2 suite-level errors, 3 errors, 77 ignored (67 assumptions)

Total time: 141 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure