You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/03/23 22:21:22 UTC
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build # 9779
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9779/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testWithin {#0 seed=[C47EB0C326693C65:B7916BB889F2C8CF]}
Error Message:
Shouldn't match I#2:Rect(minX=89.0,maxX=91.0,minY=24.0,maxY=25.0) Q:Pt(x=116.0,y=64.0)
Stack Trace:
java.lang.AssertionError: Shouldn't match I#2:Rect(minX=89.0,maxX=91.0,minY=24.0,maxY=25.0) Q:Pt(x=116.0,y=64.0)
at __randomizedtesting.SeedInfo.seed([C47EB0C326693C65:B7916BB889F2C8CF]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:358)
at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:338)
at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testWithin(SpatialOpRecursivePrefixTreeTest.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Build Log:
[...truncated 9430 lines...]
[junit4] Suite: org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(GeohashPrefixTree(maxLevels:2,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(maxLevels:1,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(maxLevels:1,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(maxLevels:1,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(GeohashPrefixTree(maxLevels:1,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 2> Ig:Rect(minX=64.0,maxX=128.0,minY=0.0,maxY=64.0) Qg:Rect(minX=64.0,maxX=128.0,minY=64.0,maxY=128.0)
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SpatialOpRecursivePrefixTreeTest -Dtests.method=testWithin -Dtests.seed=C47EB0C326693C65 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mk -Dtests.timezone=Europe/Stockholm -Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.05s J0 | SpatialOpRecursivePrefixTreeTest.testWithin {#0 seed=[C47EB0C326693C65:B7916BB889F2C8CF]} <<<
[junit4] > Throwable #1: java.lang.AssertionError: Shouldn't match I#2:Rect(minX=89.0,maxX=91.0,minY=24.0,maxY=25.0) Q:Pt(x=116.0,y=64.0)
[junit4] > at __randomizedtesting.SeedInfo.seed([C47EB0C326693C65:B7916BB889F2C8CF]:0)
[junit4] > at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:358)
[junit4] > at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:338)
[junit4] > at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testWithin(SpatialOpRecursivePrefixTreeTest.java:120)
[junit4] > at java.lang.Thread.run(Thread.java:744)
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(GeohashPrefixTree(maxLevels:3,ctx:SpatialContext.GEO)))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4] 2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=mk, timezone=Europe/Stockholm
[junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=70537768,total=130285568
[junit4] 2> NOTE: All tests run in this JVM: [SerializedStrategyTest, TestRecursivePrefixTreeStrategy, TestPointVectorStrategy, SpatialOpRecursivePrefixTreeTest]
[junit4] Completed on J0 in 1.46s, 43 tests, 1 failure <<< FAILURES!
[...truncated 20 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/build.xml:539: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1996: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/module-build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908: There were test failures: 13 suites, 74 tests, 1 failure, 2 ignored (2 assumptions)
Total time: 32 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build #
9780 - Still Failing!
Posted by Michael McCandless <lu...@mikemccandless.com>.
I like all those checks too :)
Hmm, why is SnapPuller checking whether a file exists when that file
is still being created? Is it somehow running when another thread is
already/still copying files over?
Mike McCandless
http://blog.mikemccandless.com
On Sun, Mar 23, 2014 at 9:17 PM, Mark Miller <ma...@gmail.com> wrote:
> Yeah, that's what I was thinking for "Could disable that check for Solr
> tests that replicate - not very appealing. "
>
> I like all those checks :) I suppose it's the current best option though, so
> I'll do that for now.
>
> --
> Mark Miller
> about.me/markrmiller
>
> On March 23, 2014 at 8:42:34 PM, Michael McCandless
> (lucene@mikemccandless.com) wrote:
>
> Hmmm, tricky.
>
> Could we conditionalize the check in MDW that throws IOE in openIput
> if the file is still open by createOutput? Leave it on by default but
> turn it off for these tests?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Sun, Mar 23, 2014 at 8:03 PM, Mark Miller <ma...@gmail.com> wrote:
>> Issue is that dir.openInput(fileName, IOContext.DEFAULT).close() is not a
>> perfect replacement for Directory#fileExists when using the
>> MockDirectoryWrapper - if the file is still being written, it throws an
>> IOException.
>>
>> I don't have a solution I'm in love with yet.
>>
>> Could special case it for MockDirectoryWrapper - not super appealing.
>> Might
>> be better if we make some new exceptions though.
>>
>> Could assume the file exists on IOException or IOException caused by
>> RuntimeException - not very appealing.
>>
>> Could disable that check for Solr tests that replicate - not very
>> appealing.
>>
>> Could stop using MockDirectoryWrapper on those tests - not very appealing.
>>
>> --
>> Mark Miller
>> about.me/markrmiller
>>
>> On March 23, 2014 at 7:48:47 PM, Mark Miller (markrmiller@gmail.com)
>> wrote:
>>
>> Okay, looks LUCENE-5543 related. I'll dig into it shortly.
>> --
>> Mark Miller
>> about.me/markrmiller
>>
>> On March 23, 2014 at 7:44:17 PM, Mark Miller (markrmiller@gmail.com)
>> wrote:
>>
>> Cause of this seems to be:
>>
>> 24422 T107 C8 P54679 oasc.SolrException.log ERROR SnapPull failed
>> :org.apache.solr.common.SolrException: Index fetch failed :
>> at
>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:506)
>> at
>>
>> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:324)
>> at
>>
>> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
>> at
>>
>> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:446)
>> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:234)
>> Caused by: java.io.IOException: MockDirectoryWrapper: file "_0.fdx" is
>> still
>> open for writing
>> at
>>
>> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:568)
>> at org.apache.solr.handler.SnapPuller.slowFileExists(SnapPuller.java:789)
>> at org.apache.solr.handler.SnapPuller.isIndexStale(SnapPuller.java:805)
>> at
>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:405)
>> ... 4 more
>> Caused by: java.lang.RuntimeException: unclosed IndexOutput: _0.fdx
>> at
>>
>> org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:544)
>> at
>>
>> org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:516)
>> at
>>
>> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44)
>> at
>>
>> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:109)
>> at
>>
>> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120)
>> at
>>
>> org.apache.lucene.index.StoredFieldsProcessor.initFieldsWriter(StoredFieldsProcessor.java:88)
>> at
>>
>> org.apache.lucene.index.StoredFieldsProcessor.finishDocument(StoredFieldsProcessor.java:120)
>> at
>>
>> org.apache.lucene.index.TwoStoredFieldsConsumers.finishDocument(TwoStoredFieldsConsumers.java:65)
>> at
>>
>> org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:273)
>> at
>>
>> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:271)
>> at
>>
>> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:465)
>> at
>> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1536)
>> at
>>
>> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:236)
>> at
>>
>> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
>> at
>>
>> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
>> at
>>
>> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
>> at
>>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:704)
>> at
>>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:858)
>> at
>>
>> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:557)
>> at
>>
>> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
>> at
>>
>> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
>> at
>>
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
>> at
>>
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
>> at
>>
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
>> at
>> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
>> at
>>
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
>> at
>>
>> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
>> at
>> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
>> at
>>
>> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
>> at
>>
>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
>> at
>>
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1957)
>> at
>>
>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:785)
>> at
>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:429)
>> at
>>
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
>> at
>>
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>> at
>>
>> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:134)
>> at
>>
>> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
>> at
>>
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
>> at
>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>> at
>> org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
>> at
>>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
>> at
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
>> at
>>
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
>> at
>>
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
>> at
>>
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
>> at
>>
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
>> at org.eclipse.jetty.server.Server.handle(Server.java:368)
>> at
>>
>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
>> at
>>
>> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
>> at
>>
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
>> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
>> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
>> at
>>
>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
>> at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
>> at
>>
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
>> at
>>
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
>> at
>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
>> at
>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
>> at java.lang.Thread.run(Thread.java:744)
>>
>> --
>> Mark Miller
>> about.me/markrmiller
>>
>> On March 23, 2014 at 7:04:59 PM, Policeman Jenkins Server
>> (jenkins@thetaphi.de) wrote:
>>
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9780/
>> Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
>>
>> 1 tests failed.
>> FAILED: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
>>
>> Error Message:
>> There are still nodes recoverying - waited for 90 seconds
>>
>> Stack Trace:
>> java.lang.AssertionError: There are still nodes recoverying - waited for
>> 90
>> seconds
>> at
>> __randomizedtesting.SeedInfo.seed([92A83ABF6D073DEE:134EB4A71A585DD2]:0)
>> at org.junit.Assert.fail(Assert.java:93)
>> at
>>
>> org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:177)
>> at
>>
>> org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:732)
>> at
>>
>> org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1495)
>> at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:103)
>> at
>>
>> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>>
>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
>> at
>>
>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
>> at
>>
>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
>> at
>>
>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
>> at
>>
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>>
>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>> at
>>
>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
>> at
>>
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>>
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>>
>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>> at
>>
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> at
>>
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>>
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>>
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
>> at
>>
>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
>> at
>>
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
>> at
>>
>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
>> at
>>
>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
>> at
>>
>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
>> at
>>
>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
>> at
>>
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>>
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>> at
>>
>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>> at
>>
>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>> at
>>
>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>> at
>>
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>>
>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>> at
>>
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>>
>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>> at
>>
>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>> at
>>
>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
>> at
>>
>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>> at
>>
>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at
>>
>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
>> at java.lang.Thread.run(Thread.java:744)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 10169 lines...]
>> [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
>> [junit4] 2> 80886 T677 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl
>> (false) and clientAuth (true)
>> [junit4] 2> 80887 T677 oas.BaseDistributedSearchTestCase.initHostContext
>> Setting hostContext system property: /nju/sc
>> [junit4] 2> 80892 T677 oas.SolrTestCaseJ4.setUp ###Starting
>> testDistribSearch
>> [junit4] 2> Creating dataDir:
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1395614468515
>> [junit4] 2> 80893 T677 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>> [junit4] 2> 80894 T678 oasc.ZkTestServer$ZKServerMain.runFromConfig
>> Starting
>> server
>> [junit4] 2> 80994 T677 oasc.ZkTestServer.run start zk server on port:46640
>> [junit4] 2> 80995 T677 oascc.ConnectionManager.waitForConnected Waiting
>> for
>> client to connect to ZooKeeper
>> [junit4] 2> 81204 T684 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@afcd05
>> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 81204 T677 oascc.ConnectionManager.waitForConnected Client is
>> connected to ZooKeeper
>> [junit4] 2> 81205 T677 oascc.SolrZkClient.makePath makePath: /solr
>> [junit4] 2> 81209 T677 oascc.ConnectionManager.waitForConnected Waiting
>> for
>> client to connect to ZooKeeper
>> [junit4] 2> 81210 T686 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1598a49
>> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event
>> WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 81210 T677 oascc.ConnectionManager.waitForConnected Client is
>> connected to ZooKeeper
>> [junit4] 2> 81211 T677 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1
>> [junit4] 2> 81213 T677 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/shards
>> [junit4] 2> 81215 T677 oascc.SolrZkClient.makePath makePath:
>> /collections/control_collection
>> [junit4] 2> 81216 T677 oascc.SolrZkClient.makePath makePath:
>> /collections/control_collection/shards
>> [junit4] 2> 81218 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
>> to /configs/conf1/solrconfig.xml
>> [junit4] 2> 81218 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/solrconfig.xml
>> [junit4] 2> 81220 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml
>> to /configs/conf1/schema.xml
>> [junit4] 2> 81221 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/schema.xml
>> [junit4] 2> 81224 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
>> to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>> [junit4] 2> 81224 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>> [junit4] 2> 81226 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
>> to /configs/conf1/stopwords.txt
>> [junit4] 2> 81226 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/stopwords.txt
>> [junit4] 2> 81228 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt
>> to /configs/conf1/protwords.txt
>> [junit4] 2> 81228 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/protwords.txt
>> [junit4] 2> 81230 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml
>> to /configs/conf1/currency.xml
>> [junit4] 2> 81230 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/currency.xml
>> [junit4] 2> 81231 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
>> to /configs/conf1/enumsConfig.xml
>> [junit4] 2> 81232 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/enumsConfig.xml
>> [junit4] 2> 81233 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
>> to /configs/conf1/open-exchange-rates.json
>> [junit4] 2> 81234 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/open-exchange-rates.json
>> [junit4] 2> 81235 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
>> to /configs/conf1/mapping-ISOLatin1Accent.txt
>> [junit4] 2> 81236 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/mapping-ISOLatin1Accent.txt
>> [junit4] 2> 81237 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
>> to /configs/conf1/old_synonyms.txt
>> [junit4] 2> 81238 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/old_synonyms.txt
>> [junit4] 2> 81239 T677 oasc.AbstractZkTestCase.putConfig put
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
>> to /configs/conf1/synonyms.txt
>> [junit4] 2> 81240 T677 oascc.SolrZkClient.makePath makePath:
>> /configs/conf1/synonyms.txt
>> [junit4] 2> 81475 T677 oejs.Server.doStart jetty-8.1.10.v20130312
>> [junit4] 2> 81479 T677 oejs.AbstractConnector.doStart Started
>> SelectChannelConnector@127.0.0.1:38011
>> [junit4] 2> 81480 T677 oass.SolrDispatchFilter.init
>> SolrDispatchFilter.init()
>> [junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not
>> configured for solr (NoInitialContextEx)
>> [junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome using system
>> property solr.solr.home:
>> ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864
>> [junit4] 2> 81482 T677 oasc.SolrResourceLoader.<init> new
>> SolrResourceLoader
>> for directory:
>> './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/'
>> [junit4] 2> 81536 T677 oasc.ConfigSolr.fromFile Loading container
>> configuration from
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/solr.xml
>> [junit4] 2> 81599 T677 oasc.CoreContainer.<init> New CoreContainer
>> 27496700
>> [junit4] 2> 81599 T677 oasc.CoreContainer.load Loading cores into
>> CoreContainer
>>
>> [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/]
>> [junit4] 2> 81600 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> socketTimeout to: 90000
>> [junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> urlScheme to:
>> [junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> connTimeout to: 15000
>> [junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> maxConnectionsPerHost to: 20
>> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> corePoolSize to: 0
>> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> maximumPoolSize to: 2147483647
>> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> maxThreadIdleTime to: 5
>> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> sizeOfQueue to: -1
>> [junit4] 2> 81603 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> fairnessPolicy to: false
>> [junit4] 2> 81603 T677 oasl.LogWatcher.createWatcher SLF4J impl is
>> org.slf4j.impl.Log4jLoggerFactory
>> [junit4] 2> 81604 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering
>> Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>> [junit4] 2> 81604 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
>> [junit4] 2> 81604 T677 oasc.ZkContainer.initZooKeeper Zookeeper
>> client=127.0.0.1:46640/solr
>> [junit4] 2> 81605 T677 oasc.ZkController.checkChrootPath zkHost includes
>> chroot
>> [junit4] 2> 81605 T677 oascc.ConnectionManager.waitForConnected Waiting
>> for
>> client to connect to ZooKeeper
>> [junit4] 2> 81607 T697 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1839970
>> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 81607 T677 oascc.ConnectionManager.waitForConnected Client is
>> connected to ZooKeeper
>> [junit4] 2> 81609 T677 oascc.ConnectionManager.waitForConnected Waiting
>> for
>> client to connect to ZooKeeper
>> [junit4] 2> 81609 T679 oazs.NIOServerCnxn.doIO WARN caught end of stream
>> exception EndOfStreamException: Unable to read additional data from client
>> sessionid 0x144f11ac9a60002, likely client has closed socket
>> [junit4] 2> at
>> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
>> [junit4] 2> at
>>
>> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
>> [junit4] 2> at java.lang.Thread.run(Thread.java:744)
>> [junit4] 2>
>> [junit4] 2> 81611 T699 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@6336bf
>> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event
>> WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 81611 T677 oascc.ConnectionManager.waitForConnected Client is
>> connected to ZooKeeper
>> [junit4] 2> 81612 T677 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue
>> [junit4] 2> 81614 T677 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-queue-work
>> [junit4] 2> 81616 T677 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-running
>> [junit4] 2> 81618 T677 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-completed
>> [junit4] 2> 81620 T677 oascc.SolrZkClient.makePath makePath:
>> /overseer/collection-map-failure
>> [junit4] 2> 81621 T677 oascc.SolrZkClient.makePath makePath: /live_nodes
>> [junit4] 2> 81623 T677 oasc.ZkController.createEphemeralLiveNode Register
>> node as live in ZooKeeper:/live_nodes/127.0.0.1:38011_nju%2Fsc
>> [junit4] 2> 81625 T677 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/127.0.0.1:38011_nju%2Fsc
>> [junit4] 2> 81627 T677 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect
>> [junit4] 2> 81628 T677 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/election
>> [junit4] 2> 81631 T677 oasc.OverseerElectionContext.runLeaderProcess I am
>> going to be the leader 127.0.0.1:38011_nju%2Fsc
>> [junit4] 2> 81631 T677 oascc.SolrZkClient.makePath makePath:
>> /overseer_elect/leader
>> [junit4] 2> 81632 T677 oasc.Overseer.start Overseer
>> (id=91462989808795651-127.0.0.1:38011_nju%2Fsc-n_0000000000) starting
>> [junit4] 2> 81637 T677 oascc.SolrZkClient.makePath makePath:
>> /overseer/queue-work
>> [junit4] 2> 81645 T701 oasc.OverseerCollectionProcessor.run Process
>> current
>> queue of collection creations
>> [junit4] 2> 81645 T677 oascc.SolrZkClient.makePath makePath:
>> /clusterstate.json
>> [junit4] 2> 81646 T701
>> oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing
>> overseer nodes
>> [junit4] 2> 81647 T677 oascc.SolrZkClient.makePath makePath: /aliases.json
>> [junit4] 2> 81648 T677
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 81650 T700 oasc.Overseer$ClusterStateUpdater.run Starting to
>> work on the main queue
>> [junit4] 2> 81655 T702 oasc.ZkController.publish publishing
>> core=collection1
>> state=down collection=control_collection
>> [junit4] 2> 81656 T702 oasc.ZkController.publish numShards not found on
>> descriptor - reading it from system property
>> [junit4] 2> 81657 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 81657 T702 oasc.ZkController.waitForCoreNodeName look for our
>> core node name
>> [junit4] 2> 81658 T700 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState
>> Update
>> state numShards=1 message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"control_collection",
>> [junit4] 2> "numShards":"1",
>> [junit4] 2> "core_node_name":null}
>> [junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.createCollection
>> Create collection control_collection with shards [shard1]
>> [junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState
>> Assigning new node to shard shard=shard1
>> [junit4] 2> 81661 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 81662 T699 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 82658 T702 oasc.ZkController.waitForShardId waiting to find
>> shard id in clusterstate for collection1
>> [junit4] 2> 82658 T702 oasc.CoreContainer.create Creating SolrCore
>> 'collection1' using instanceDir:
>>
>> ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1
>> [junit4] 2> 82658 T702 oasc.ZkController.createCollectionZkNode Check for
>> collection zkNode:control_collection
>> [junit4] 2> 82659 T702 oasc.ZkController.createCollectionZkNode Collection
>> zkNode exists
>> [junit4] 2> 82659 T702 oascc.ZkStateReader.readConfigName Load collection
>> config from:/collections/control_collection
>> [junit4] 2> 82660 T702 oascc.ZkStateReader.readConfigName
>> path=/collections/control_collection configName=conf1 specified config
>> exists in ZooKeeper
>> [junit4] 2> 82660 T702 oasc.SolrResourceLoader.<init> new
>> SolrResourceLoader
>> for directory:
>>
>> './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/'
>> [junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding
>>
>> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/.svn/'
>> to classloader
>> [junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding
>>
>> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/classes/'
>> to classloader
>> [junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding
>>
>> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/README'
>> to classloader
>> [junit4] 2> 82714 T702 oasc.SolrConfig.<init> Using Lucene MatchVersion:
>> LUCENE_48
>> [junit4] 2> 82762 T702 oasc.SolrConfig.<init> Loaded SolrConfig:
>> solrconfig.xml
>> [junit4] 2> 82763 T702 oass.IndexSchema.readSchema Reading Solr Schema
>> from
>> schema.xml
>> [junit4] 2> 82788 T702 oass.IndexSchema.readSchema [collection1] Schema
>> name=test
>> [junit4] 2> 83255 T702 oass.IndexSchema.readSchema default search field in
>> schema is text
>> [junit4] 2> 83258 T702 oass.IndexSchema.readSchema unique key field: id
>> [junit4] 2> 83260 T702 oass.FileExchangeRateProvider.reload Reloading
>> exchange rates from file currency.xml
>> [junit4] 2> 83264 T702 oass.FileExchangeRateProvider.reload Reloading
>> exchange rates from file currency.xml
>> [junit4] 2> 83285 T702 oasc.SolrCore.initDirectoryFactory
>> org.apache.solr.core.MockDirectoryFactory
>> [junit4] 2> 83286 T702 oasc.SolrCore.<init> [collection1] Opening new
>> SolrCore at
>>
>> ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/,
>> dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/
>> [junit4] 2> 83289 T702 oasc.JmxMonitoredMap.<init> JMX monitoring is
>> enabled. Adding Solr mbeans to JMX Server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
>> [junit4] 2> 83292 T702 oasc.CachingDirectoryFactory.get return new
>> directory
>> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data
>> [junit4] 2> 83292 T702 oasc.SolrCore.getNewIndexDir New index directory
>> detected: old=null
>>
>> new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index/
>> [junit4] 2> 83292 T702 oasc.SolrCore.initIndex WARN [collection1] Solr
>> index
>> directory
>> './org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index'
>> doesn't exist. Creating new index...
>> [junit4] 2> 83293 T702 oasc.CachingDirectoryFactory.get return new
>> directory
>> for
>> ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index
>> [junit4] 2> 83293 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy
>> wrapping class org.apache.lucene.index.LogDocMergePolicy:
>> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39,
>> maxMergeSize=9223372036854775807,
>> maxMergeSizeForForcedMerge=9223372036854775807,
>> calibrateSizeByDeletes=true,
>> maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
>> noCFSRatio=1.0]
>> [junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.onCommit
>> SolrDeletionPolicy.onCommit: commits: num=1
>> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d
>>
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
>> [junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.updateCommits newest commit
>> generation = 1
>> [junit4] 2> 83299 T702 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "nodistrib"
>> [junit4] 2> 83310 T702 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "dedupe"
>> [junit4] 2> 83313 T702 oasup.UpdateRequestProcessorChain.init inserting
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain
>> "dedupe"
>> [junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "stored_sig"
>> [junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init inserting
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain
>> "stored_sig"
>> [junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>> [junit4] 2> 83338 T702 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>> [junit4] 2> 83339 T702 oasup.UpdateRequestProcessorChain.init inserting
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain
>> "distrib-dup-test-chain-implicit"
>> [junit4] 2> 83339 T702 oasc.SolrCore.loadUpdateProcessorChains no
>> updateRequestProcessorChain defined as default, creating implicit default
>> [junit4] 2> 83341 T702 oasc.RequestHandlers.initHandlersFromConfig created
>> standard: solr.StandardRequestHandler
>> [junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig created
>> /get: solr.RealTimeGetHandler
>> [junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig adding
>> lazy requestHandler: solr.ReplicationHandler
>> [junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created
>> /replication: solr.ReplicationHandler
>> [junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created
>> /update: solr.UpdateRequestHandler
>> [junit4] 2> 83344 T702 oasc.RequestHandlers.initHandlersFromConfig created
>> /admin/: org.apache.solr.handler.admin.AdminHandlers
>> [junit4] 2> 83349 T702 oasc.RequestHandlers.initHandlersFromConfig created
>> /admin/file: solr.admin.ShowFileRequestHandler
>> [junit4] 2> 83351 T702 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 83355 T702 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>> [junit4] 2> 83355 T702 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>> [junit4] 2> 83357 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy
>> wrapping class org.apache.lucene.index.LogByteSizeMergePolicy:
>> [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41,
>> maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807,
>> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
>> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>> [junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.onInit
>> SolrDeletionPolicy.onInit: commits: num=1
>> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d
>>
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
>> [junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.updateCommits newest commit
>> generation = 1
>> [junit4] 2> 83358 T702 oass.SolrIndexSearcher.<init> Opening
>> Searcher@1c23829[collection1] main
>> [junit4] 2> 83365 T703 oasc.SolrCore.registerSearcher [collection1]
>> Registered new searcher Searcher@1c23829[collection1]
>> main{StandardDirectoryReader(segments_1:1:nrt)}
>> [junit4] 2> 83368 T702 oasc.CoreContainer.registerCore registering core:
>> collection1
>> [junit4] 2> 83369 T677 oass.SolrDispatchFilter.init
>>
>> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
>> [junit4] 2> 83369 T677 oass.SolrDispatchFilter.init
>> SolrDispatchFilter.init() done
>> [junit4] 2> 83369 T706 oasc.ZkController.register Register replica -
>> core:collection1 address:http://127.0.0.1:38011/nju/sc
>> collection:control_collection shard:shard1
>> [junit4] 2> 83370 T677 oascc.ConnectionManager.waitForConnected Waiting
>> for
>> client to connect to ZooKeeper
>> [junit4] 2> 83371 T706 oascc.SolrZkClient.makePath makePath:
>> /collections/control_collection/leader_elect/shard1/election
>> [junit4] 2> 83372 T708 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@9657d4
>> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event
>> WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 83372 T677 oascc.ConnectionManager.waitForConnected Client is
>> connected to ZooKeeper
>> [junit4] 2> 83373 T677
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 83375 T677 oasc.ChaosMonkey.monkeyLog monkey: init - expire
>> sessions:false cause connection loss:false
>> [junit4] 2> 83376 T706 oasc.ShardLeaderElectionContext.runLeaderProcess
>> Running the leader process for shard shard1
>> [junit4] 2> 83377 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 83378 T706
>> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
>> found to continue.
>> [junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I
>> may be the new leader - try and sync
>> [junit4] 2> ASYNC NEW_CORE C596 name=collection1
>> org.apache.solr.core.SolrCore@69814a
>> url=http://127.0.0.1:38011/nju/sc/collection1
>> node=127.0.0.1:38011_nju%2Fsc
>> C596_STATE=coll:control_collection core:collection1 props:{state=down,
>> base_url=http://127.0.0.1:38011/nju/sc, core=collection1,
>> node_name=127.0.0.1:38011_nju%2Fsc}
>> [junit4] 2> 83378 T706 C596 P38011 oasc.SyncStrategy.sync Sync replicas to
>> http://127.0.0.1:38011/nju/sc/collection1/
>> [junit4] 2> 83379 T700 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncReplicas Sync
>> Success - now sync replicas to me
>> [junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncToMe
>> http://127.0.0.1:38011/nju/sc/collection1/ has no replicas
>> [junit4] 2> 83379 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I
>> am
>> the new leader: http://127.0.0.1:38011/nju/sc/collection1/ shard1
>> [junit4] 2> 83380 T706 oascc.SolrZkClient.makePath makePath:
>> /collections/control_collection/leaders/shard1
>> [junit4] 2> 83381 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 83382 T699 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 83382 T708 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 83385 T700 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 83386 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 83490 T699 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 83498 T708 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 83539 T706 oasc.ZkController.register We are
>> http://127.0.0.1:38011/nju/sc/collection1/ and leader is
>> http://127.0.0.1:38011/nju/sc/collection1/
>> [junit4] 2> 83539 T706 oasc.ZkController.register No LogReplay needed for
>> core=collection1 baseURL=http://127.0.0.1:38011/nju/sc
>> [junit4] 2> 83539 T706 oasc.ZkController.checkRecovery I am the leader, no
>> recovery necessary
>> [junit4] 2> 83540 T706 oasc.ZkController.publish publishing
>> core=collection1
>> state=active collection=control_collection
>> [junit4] 2> 83540 T706 oasc.ZkController.publish numShards not found on
>> descriptor - reading it from system property
>> [junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 83542 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 83542 T706 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 83543 T700 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 83544 T700 oasc.Overseer$ClusterStateUpdater.updateState
>> Update
>> state numShards=1 message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
>> [junit4] 2> "shard":"shard1",
>> [junit4] 2> "collection":"control_collection",
>> [junit4] 2> "numShards":"1",
>> [junit4] 2> "core_node_name":"core_node1"}
>> [junit4] 2> 83546 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 83631 T677 oasc.AbstractFullDistribZkTestBase.createJettys
>> create jetty 1
>> [junit4] 2> 83631 T677 oejs.Server.doStart jetty-8.1.10.v20130312
>> [junit4] 2> 83651 T677 oejs.AbstractConnector.doStart Started
>> SelectChannelConnector@127.0.0.1:39738
>> [junit4] 2> 83652 T677 oass.SolrDispatchFilter.init
>> SolrDispatchFilter.init()
>> [junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not
>> configured for solr (NoInitialContextEx)
>> [junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome using system
>> property solr.solr.home:
>>
>> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998
>> [junit4] 2> 83655 T699 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 83655 T677 oasc.SolrResourceLoader.<init> new
>> SolrResourceLoader
>> for directory:
>>
>> '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/'
>> [junit4] 2> 83655 T708 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 83669 T677 oasc.ConfigSolr.fromFile Loading container
>> configuration from
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/solr.xml
>> [junit4] 2> 83720 T677 oasc.CoreContainer.<init> New CoreContainer
>> 29042762
>> [junit4] 2> 83721 T677 oasc.CoreContainer.load Loading cores into
>> CoreContainer
>>
>> [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/]
>> [junit4] 2> 83722 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> socketTimeout to: 90000
>> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> urlScheme to:
>> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> connTimeout to: 15000
>> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> maxConnectionsPerHost to: 20
>> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> corePoolSize to: 0
>> [junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> maximumPoolSize to: 2147483647
>> [junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> maxThreadIdleTime to: 5
>> [junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> sizeOfQueue to: -1
>> [junit4] 2> 83725 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> fairnessPolicy to: false
>> [junit4] 2> 83725 T677 oasl.LogWatcher.createWatcher SLF4J impl is
>> org.slf4j.impl.Log4jLoggerFactory
>> [junit4] 2> 83726 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering
>> Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>> [junit4] 2> 83726 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
>> [junit4] 2> 83726 T677 oasc.ZkContainer.initZooKeeper Zookeeper
>> client=127.0.0.1:46640/solr
>> [junit4] 2> 83727 T677 oasc.ZkController.checkChrootPath zkHost includes
>> chroot
>> [junit4] 2> 83727 T677 oascc.ConnectionManager.waitForConnected Waiting
>> for
>> client to connect to ZooKeeper
>> [junit4] 2> 83729 T719 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@5231ef
>> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 83729 T677 oascc.ConnectionManager.waitForConnected Client is
>> connected to ZooKeeper
>> [junit4] 2> 83731 T677 oascc.ConnectionManager.waitForConnected Waiting
>> for
>> client to connect to ZooKeeper
>> [junit4] 2> 83732 T721 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@1075f95
>> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event
>> WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 83733 T677 oascc.ConnectionManager.waitForConnected Client is
>> connected to ZooKeeper
>> [junit4] 2> 83737 T677
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 84742 T677 oasc.ZkController.createEphemeralLiveNode Register
>> node as live in ZooKeeper:/live_nodes/127.0.0.1:39738_nju%2Fsc
>> [junit4] 2> 84743 T677 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/127.0.0.1:39738_nju%2Fsc
>> [junit4] 2> 84745 T708 oascc.ZkStateReader$3.process Updating live
>> nodes...
>> (2)
>> [junit4] 2> 84745 T721 oascc.ZkStateReader$3.process Updating live
>> nodes...
>> (2)
>> [junit4] 2> 84745 T699 oascc.ZkStateReader$3.process Updating live
>> nodes...
>> (2)
>> [junit4] 2> 84751 T722 oasc.ZkController.publish publishing
>> core=collection1
>> state=down collection=collection1
>> [junit4] 2> 84752 T722 oasc.ZkController.publish numShards not found on
>> descriptor - reading it from system property
>> [junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 84755 T722 oasc.ZkController.waitForCoreNodeName look for our
>> core node name
>> [junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 84756 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 84757 T700 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.updateState
>> Update
>> state numShards=1 message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"1",
>> [junit4] 2> "core_node_name":null}
>> [junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.createCollection
>> Create collection collection1 with shards [shard1]
>> [junit4] 2> 84758 T700 oasc.Overseer$ClusterStateUpdater.updateState
>> Assigning new node to shard shard=shard1
>> [junit4] 2> 84759 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 84862 T721 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 84862 T699 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 84862 T708 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 85756 T722 oasc.ZkController.waitForShardId waiting to find
>> shard id in clusterstate for collection1
>> [junit4] 2> 85756 T722 oasc.CoreContainer.create Creating SolrCore
>> 'collection1' using instanceDir:
>>
>> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1
>> [junit4] 2> 85756 T722 oasc.ZkController.createCollectionZkNode Check for
>> collection zkNode:collection1
>> [junit4] 2> 85757 T722 oasc.ZkController.createCollectionZkNode Collection
>> zkNode exists
>> [junit4] 2> 85758 T722 oascc.ZkStateReader.readConfigName Load collection
>> config from:/collections/collection1
>> [junit4] 2> 85759 T722 oascc.ZkStateReader.readConfigName
>> path=/collections/collection1 configName=conf1 specified config exists in
>> ZooKeeper
>> [junit4] 2> 85759 T722 oasc.SolrResourceLoader.<init> new
>> SolrResourceLoader
>> for directory:
>>
>> '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/'
>> [junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding
>>
>> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/.svn/'
>> to classloader
>> [junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding
>>
>> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/classes/'
>> to classloader
>> [junit4] 2> 85761 T722 oasc.SolrResourceLoader.replaceClassLoader Adding
>>
>> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/README'
>> to classloader
>> [junit4] 2> 85807 T722 oasc.SolrConfig.<init> Using Lucene MatchVersion:
>> LUCENE_48
>> [junit4] 2> 85855 T722 oasc.SolrConfig.<init> Loaded SolrConfig:
>> solrconfig.xml
>> [junit4] 2> 85856 T722 oass.IndexSchema.readSchema Reading Solr Schema
>> from
>> schema.xml
>> [junit4] 2> 85866 T722 oass.IndexSchema.readSchema [collection1] Schema
>> name=test
>> [junit4] 2> 86312 T722 oass.IndexSchema.readSchema default search field in
>> schema is text
>> [junit4] 2> 86315 T722 oass.IndexSchema.readSchema unique key field: id
>> [junit4] 2> 86317 T722 oass.FileExchangeRateProvider.reload Reloading
>> exchange rates from file currency.xml
>> [junit4] 2> 86321 T722 oass.FileExchangeRateProvider.reload Reloading
>> exchange rates from file currency.xml
>> [junit4] 2> 86342 T722 oasc.SolrCore.initDirectoryFactory
>> org.apache.solr.core.MockDirectoryFactory
>> [junit4] 2> 86343 T722 oasc.SolrCore.<init> [collection1] Opening new
>> SolrCore at
>>
>> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/,
>> dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/
>> [junit4] 2> 86343 T722 oasc.JmxMonitoredMap.<init> JMX monitoring is
>> enabled. Adding Solr mbeans to JMX Server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
>> [junit4] 2> 86346 T722 oasc.CachingDirectoryFactory.get return new
>> directory
>> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1
>> [junit4] 2> 86346 T722 oasc.SolrCore.getNewIndexDir New index directory
>> detected: old=null
>> new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index/
>> [junit4] 2> 86346 T722 oasc.SolrCore.initIndex WARN [collection1] Solr
>> index
>> directory
>> './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index'
>> doesn't
>> exist. Creating new index...
>> [junit4] 2> 86347 T722 oasc.CachingDirectoryFactory.get return new
>> directory
>> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index
>> [junit4] 2> 86347 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy
>> wrapping class org.apache.lucene.index.LogDocMergePolicy:
>> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39,
>> maxMergeSize=9223372036854775807,
>> maxMergeSizeForForcedMerge=9223372036854775807,
>> calibrateSizeByDeletes=true,
>> maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
>> noCFSRatio=1.0]
>> [junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.onCommit
>> SolrDeletionPolicy.onCommit: commits: num=1
>> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f
>>
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
>> [junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.updateCommits newest commit
>> generation = 1
>> [junit4] 2> 86353 T722 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "nodistrib"
>> [junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "dedupe"
>> [junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init inserting
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain
>> "dedupe"
>> [junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "stored_sig"
>> [junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init inserting
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain
>> "stored_sig"
>> [junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>> [junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>> [junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init inserting
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain
>> "distrib-dup-test-chain-implicit"
>> [junit4] 2> 86356 T722 oasc.SolrCore.loadUpdateProcessorChains no
>> updateRequestProcessorChain defined as default, creating implicit default
>> [junit4] 2> 86357 T722 oasc.RequestHandlers.initHandlersFromConfig created
>> standard: solr.StandardRequestHandler
>> [junit4] 2> 86358 T722 oasc.RequestHandlers.initHandlersFromConfig created
>> /get: solr.RealTimeGetHandler
>> [junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig adding
>> lazy requestHandler: solr.ReplicationHandler
>> [junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig created
>> /replication: solr.ReplicationHandler
>> [junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created
>> /update: solr.UpdateRequestHandler
>> [junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created
>> /admin/: org.apache.solr.handler.admin.AdminHandlers
>> [junit4] 2> 86361 T722 oasc.RequestHandlers.initHandlersFromConfig created
>> /admin/file: solr.admin.ShowFileRequestHandler
>> [junit4] 2> 86364 T722 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 86367 T722 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>> [junit4] 2> 86367 T722 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>> [junit4] 2> 86368 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy
>> wrapping class org.apache.lucene.index.LogByteSizeMergePolicy:
>> [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41,
>> maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807,
>> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
>> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>> [junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.onInit
>> SolrDeletionPolicy.onInit: commits: num=1
>> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f
>>
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
>> [junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.updateCommits newest commit
>> generation = 1
>> [junit4] 2> 86369 T722 oass.SolrIndexSearcher.<init> Opening
>> Searcher@383c54[collection1] main
>> [junit4] 2> 86376 T723 oasc.SolrCore.registerSearcher [collection1]
>> Registered new searcher Searcher@383c54[collection1]
>> main{StandardDirectoryReader(segments_1:1:nrt)}
>> [junit4] 2> 86381 T722 oasc.CoreContainer.registerCore registering core:
>> collection1
>> [junit4] 2> 86382 T677 oass.SolrDispatchFilter.init
>>
>> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
>> [junit4] 2> 86382 T677 oass.SolrDispatchFilter.init
>> SolrDispatchFilter.init() done
>> [junit4] 2> 86382 T726 oasc.ZkController.register Register replica -
>> core:collection1 address:http://127.0.0.1:39738/nju/sc
>> collection:collection1 shard:shard1
>> [junit4] 2> 86383 T726 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leader_elect/shard1/election
>> [junit4] 2> 86387 T726 oasc.ShardLeaderElectionContext.runLeaderProcess
>> Running the leader process for shard shard1
>> [junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 86389 T726
>> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
>> found to continue.
>> [junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I
>> may be the new leader - try and sync
>> [junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> ASYNC NEW_CORE C597 name=collection1
>> org.apache.solr.core.SolrCore@a6a2f3
>> url=http://127.0.0.1:39738/nju/sc/collection1
>> node=127.0.0.1:39738_nju%2Fsc
>> C597_STATE=coll:collection1 core:collection1 props:{state=down,
>> base_url=http://127.0.0.1:39738/nju/sc, core=collection1,
>> node_name=127.0.0.1:39738_nju%2Fsc}
>> [junit4] 2> 86389 T726 C597 P39738 oasc.SyncStrategy.sync Sync replicas to
>> http://127.0.0.1:39738/nju/sc/collection1/
>> [junit4] 2> 86390 T700 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 86390 T726 C597 P39738 oasc.SyncStrategy.syncReplicas Sync
>> Success - now sync replicas to me
>> [junit4] 2> 86391 T726 C597 P39738 oasc.SyncStrategy.syncToMe
>> http://127.0.0.1:39738/nju/sc/collection1/ has no replicas
>> [junit4] 2> 86390 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 86391 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I
>> am
>> the new leader: http://127.0.0.1:39738/nju/sc/collection1/ shard1
>> [junit4] 2> 86391 T726 oascc.SolrZkClient.makePath makePath:
>> /collections/collection1/leaders/shard1
>> [junit4] 2> 86392 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 86394 T699 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 86394 T721 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 86394 T708 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 86397 T700 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 86399 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 86501 T708 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 86501 T721 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 86501 T699 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 86547 T726 oasc.ZkController.register We are
>> http://127.0.0.1:39738/nju/sc/collection1/ and leader is
>> http://127.0.0.1:39738/nju/sc/collection1/
>> [junit4] 2> 86547 T726 oasc.ZkController.register No LogReplay needed for
>> core=collection1 baseURL=http://127.0.0.1:39738/nju/sc
>> [junit4] 2> 86547 T726 oasc.ZkController.checkRecovery I am the leader, no
>> recovery necessary
>> [junit4] 2> 86548 T726 oasc.ZkController.publish publishing
>> core=collection1
>> state=active collection=collection1
>> [junit4] 2> 86548 T726 oasc.ZkController.publish numShards not found on
>> descriptor - reading it from system property
>> [junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 86550 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 86550 T726 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 86551 T700 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 86551 T700 oasc.Overseer$ClusterStateUpdater.updateState
>> Update
>> state numShards=1 message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"active",
>> [junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
>> [junit4] 2> "shard":"shard1",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"1",
>> [junit4] 2> "core_node_name":"core_node1"}
>> [junit4] 2> 86553 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 86655 T699 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 86655 T708 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 86655 T721 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>> [junit4] 2> 86667 T677 oasc.AbstractFullDistribZkTestBase.createJettys
>> create jetty 2
>> [junit4] 2> 86668 T677 oejs.Server.doStart jetty-8.1.10.v20130312
>> [junit4] 2> 86670 T677 oejs.AbstractConnector.doStart Started
>> SelectChannelConnector@127.0.0.1:40129
>> [junit4] 2> 86671 T677 oass.SolrDispatchFilter.init
>> SolrDispatchFilter.init()
>> [junit4] 2> 86671 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not
>> configured for solr (NoInitialContextEx)
>> [junit4] 2> 86672 T677 oasc.SolrResourceLoader.locateSolrHome using system
>> property solr.solr.home:
>> ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005
>> [junit4] 2> 86672 T677 oasc.SolrResourceLoader.<init> new
>> SolrResourceLoader
>> for directory:
>> './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/'
>> [junit4] 2> 86704 T677 oasc.ConfigSolr.fromFile Loading container
>> configuration from
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/solr.xml
>> [junit4] 2> 86751 T677 oasc.CoreContainer.<init> New CoreContainer
>> 10907151
>> [junit4] 2> 86751 T677 oasc.CoreContainer.load Loading cores into
>> CoreContainer
>> [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/]
>> [junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> socketTimeout to: 90000
>> [junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> urlScheme to:
>> [junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> connTimeout to: 15000
>> [junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> maxConnectionsPerHost to: 20
>> [junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> corePoolSize to: 0
>> [junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> maximumPoolSize to: 2147483647
>> [junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> maxThreadIdleTime to: 5
>> [junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> sizeOfQueue to: -1
>> [junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting
>> fairnessPolicy to: false
>> [junit4] 2> 86756 T677 oasl.LogWatcher.createWatcher SLF4J impl is
>> org.slf4j.impl.Log4jLoggerFactory
>> [junit4] 2> 86756 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering
>> Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>> [junit4] 2> 86756 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
>> [junit4] 2> 86757 T677 oasc.ZkContainer.initZooKeeper Zookeeper
>> client=127.0.0.1:46640/solr
>> [junit4] 2> 86757 T677 oasc.ZkController.checkChrootPath zkHost includes
>> chroot
>> [junit4] 2> 86758 T677 oascc.ConnectionManager.waitForConnected Waiting
>> for
>> client to connect to ZooKeeper
>> [junit4] 2> 86759 T737 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@83c633
>> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 86760 T677 oascc.ConnectionManager.waitForConnected Client is
>> connected to ZooKeeper
>> [junit4] 2> 86762 T677 oascc.ConnectionManager.waitForConnected Waiting
>> for
>> client to connect to ZooKeeper
>> [junit4] 2> 86763 T739 oascc.ConnectionManager.process Watcher
>> org.apache.solr.common.cloud.ConnectionManager@18288ea
>> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event
>> WatchedEvent
>> state:SyncConnected type:None path:null path:null type:None
>> [junit4] 2> 86763 T677 oascc.ConnectionManager.waitForConnected Client is
>> connected to ZooKeeper
>> [junit4] 2> 86768 T677
>> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
>> state from ZooKeeper...
>> [junit4] 2> 87770 T677 oasc.ZkController.createEphemeralLiveNode Register
>> node as live in ZooKeeper:/live_nodes/127.0.0.1:40129_nju%2Fsc
>> [junit4] 2> 87771 T677 oascc.SolrZkClient.makePath makePath:
>> /live_nodes/127.0.0.1:40129_nju%2Fsc
>> [junit4] 2> 87773 T699 oascc.ZkStateReader$3.process Updating live
>> nodes...
>> (3)
>> [junit4] 2> 87773 T708 oascc.ZkStateReader$3.process Updating live
>> nodes...
>> (3)
>> [junit4] 2> 87773 T721 oascc.ZkStateReader$3.process Updating live
>> nodes...
>> (3)
>> [junit4] 2> 87774 T739 oascc.ZkStateReader$3.process Updating live
>> nodes...
>> (3)
>> [junit4] 2> 87781 T740 oasc.ZkController.publish publishing
>> core=collection1
>> state=down collection=collection1
>> [junit4] 2> 87781 T740 oasc.ZkController.publish numShards not found on
>> descriptor - reading it from system property
>> [junit4] 2> 87783 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 87783 T740 oasc.ZkController.waitForCoreNodeName look for our
>> core node name
>> [junit4] 2> 87785 T700 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState
>> Update
>> state numShards=1 message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"down",
>> [junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
>> [junit4] 2> "shard":null,
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"1",
>> [junit4] 2> "core_node_name":null}
>> [junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState
>> Collection already exists with numShards=1
>> [junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState
>> Assigning new node to shard shard=shard1
>> [junit4] 2> 87789 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 87890 T739 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>> [junit4] 2> 87890 T708 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>> [junit4] 2> 87890 T699 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>> [junit4] 2> 87890 T721 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>> [junit4] 2> 88785 T740 oasc.ZkController.waitForShardId waiting to find
>> shard id in clusterstate for collection1
>> [junit4] 2> 88785 T740 oasc.CoreContainer.create Creating SolrCore
>> 'collection1' using instanceDir:
>> ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1
>> [junit4] 2> 88785 T740 oasc.ZkController.createCollectionZkNode Check for
>> collection zkNode:collection1
>> [junit4] 2> 88786 T740 oasc.ZkController.createCollectionZkNode Collection
>> zkNode exists
>> [junit4] 2> 88786 T740 oascc.ZkStateReader.readConfigName Load collection
>> config from:/collections/collection1
>> [junit4] 2> 88787 T740 oascc.ZkStateReader.readConfigName
>> path=/collections/collection1 configName=conf1 specified config exists in
>> ZooKeeper
>> [junit4] 2> 88787 T740 oasc.SolrResourceLoader.<init> new
>> SolrResourceLoader
>> for directory:
>> './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/'
>> [junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding
>>
>> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/.svn/'
>> to classloader
>> [junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding
>>
>> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/classes/'
>> to classloader
>> [junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding
>>
>> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/README'
>> to classloader
>> [junit4] 2> 88833 T740 oasc.SolrConfig.<init> Using Lucene MatchVersion:
>> LUCENE_48
>> [junit4] 2> 88882 T740 oasc.SolrConfig.<init> Loaded SolrConfig:
>> solrconfig.xml
>> [junit4] 2> 88883 T740 oass.IndexSchema.readSchema Reading Solr Schema
>> from
>> schema.xml
>> [junit4] 2> 88895 T740 oass.IndexSchema.readSchema [collection1] Schema
>> name=test
>> [junit4] 2> 89367 T740 oass.IndexSchema.readSchema default search field in
>> schema is text
>> [junit4] 2> 89370 T740 oass.IndexSchema.readSchema unique key field: id
>> [junit4] 2> 89372 T740 oass.FileExchangeRateProvider.reload Reloading
>> exchange rates from file currency.xml
>> [junit4] 2> 89375 T740 oass.FileExchangeRateProvider.reload Reloading
>> exchange rates from file currency.xml
>> [junit4] 2> 89392 T740 oasc.SolrCore.initDirectoryFactory
>> org.apache.solr.core.MockDirectoryFactory
>> [junit4] 2> 89393 T740 oasc.SolrCore.<init> [collection1] Opening new
>> SolrCore at
>> ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/,
>> dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/
>> [junit4] 2> 89393 T740 oasc.JmxMonitoredMap.<init> JMX monitoring is
>> enabled. Adding Solr mbeans to JMX Server:
>> com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
>> [junit4] 2> 89394 T740 oasc.CachingDirectoryFactory.get return new
>> directory
>> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
>> [junit4] 2> 89395 T740 oasc.SolrCore.getNewIndexDir New index directory
>> detected: old=null
>> new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index/
>> [junit4] 2> 89395 T740 oasc.SolrCore.initIndex WARN [collection1] Solr
>> index
>> directory
>> './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index'
>> doesn't
>> exist. Creating new index...
>> [junit4] 2> 89395 T740 oasc.CachingDirectoryFactory.get return new
>> directory
>> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
>> [junit4] 2> 89396 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy
>> wrapping class org.apache.lucene.index.LogDocMergePolicy:
>> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39,
>> maxMergeSize=9223372036854775807,
>> maxMergeSizeForForcedMerge=9223372036854775807,
>> calibrateSizeByDeletes=true,
>> maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
>> noCFSRatio=1.0]
>> [junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.onCommit
>> SolrDeletionPolicy.onCommit: commits: num=1
>> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d
>>
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
>> [junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.updateCommits newest commit
>> generation = 1
>> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "nodistrib"
>> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "dedupe"
>> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init inserting
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain
>> "dedupe"
>> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "stored_sig"
>> [junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init inserting
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain
>> "stored_sig"
>> [junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>> [junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating
>> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>> [junit4] 2> 89404 T740 oasup.UpdateRequestProcessorChain.init inserting
>> DistributedUpdateProcessorFactory into updateRequestProcessorChain
>> "distrib-dup-test-chain-implicit"
>> [junit4] 2> 89404 T740 oasc.SolrCore.loadUpdateProcessorChains no
>> updateRequestProcessorChain defined as default, creating implicit default
>> [junit4] 2> 89405 T740 oasc.RequestHandlers.initHandlersFromConfig created
>> standard: solr.StandardRequestHandler
>> [junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created
>> /get: solr.RealTimeGetHandler
>> [junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig adding
>> lazy requestHandler: solr.ReplicationHandler
>> [junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created
>> /replication: solr.ReplicationHandler
>> [junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created
>> /update: solr.UpdateRequestHandler
>> [junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created
>> /admin/: org.apache.solr.handler.admin.AdminHandlers
>> [junit4] 2> 89408 T740 oasc.RequestHandlers.initHandlersFromConfig created
>> /admin/file: solr.admin.ShowFileRequestHandler
>> [junit4] 2> 89411 T740 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
>> [junit4] 2> 89414 T740 oasu.CommitTracker.<init> Hard AutoCommit: disabled
>> [junit4] 2> 89414 T740 oasu.CommitTracker.<init> Soft AutoCommit: disabled
>> [junit4] 2> 89415 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy
>> wrapping class org.apache.lucene.index.LogByteSizeMergePolicy:
>> [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41,
>> maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807,
>> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
>> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>> [junit4] 2> 89415 T740 oasc.SolrDeletionPolicy.onInit
>> SolrDeletionPolicy.onInit: commits: num=1
>> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d
>>
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
>> [junit4] 2> 89416 T740 oasc.SolrDeletionPolicy.updateCommits newest commit
>> generation = 1
>> [junit4] 2> 89416 T740 oass.SolrIndexSearcher.<init> Opening
>> Searcher@c599fa[collection1] main
>> [junit4] 2> 89422 T741 oasc.SolrCore.registerSearcher [collection1]
>> Registered new searcher Searcher@c599fa[collection1]
>> main{StandardDirectoryReader(segments_1:1:nrt)}
>> [junit4] 2> 89426 T740 oasc.CoreContainer.registerCore registering core:
>> collection1
>> [junit4] 2> 89426 T677 oass.SolrDispatchFilter.init
>>
>> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
>> [junit4] 2> 89427 T677 oass.SolrDispatchFilter.init
>> SolrDispatchFilter.init() done
>> [junit4] 2> 89426 T744 oasc.ZkController.register Register replica -
>> core:collection1 address:http://127.0.0.1:40129/nju/sc
>> collection:collection1 shard:shard1
>> [junit4] 2> 89428 T677 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 89430 T744 oasc.ZkController.register We are
>> http://127.0.0.1:40129/nju/sc/collection1/ and leader is
>> http://127.0.0.1:39738/nju/sc/collection1/
>> [junit4] 2> 89430 T744 oasc.ZkController.register No LogReplay needed for
>> core=collection1 baseURL=http://127.0.0.1:40129/nju/sc
>> [junit4] 2> 89431 T744 oasc.ZkController.checkRecovery Core needs to
>> recover:collection1
>> [junit4] 2> 89431 T744 oasu.DefaultSolrCoreState.doRecovery Running
>> recovery
>> - first canceling any ongoing recovery
>> [junit4] 2> 89431 T744 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> ASYNC NEW_CORE C598 name=collection1
>> org.apache.solr.core.SolrCore@1ec0a26
>> url=http://127.0.0.1:40129/nju/sc/collection1
>> node=127.0.0.1:40129_nju%2Fsc
>> C598_STATE=coll:collection1 core:collection1 props:{state=down,
>> base_url=http://127.0.0.1:40129/nju/sc, core=collection1,
>> node_name=127.0.0.1:40129_nju%2Fsc}
>> [junit4] 2> 89432 T747 C598 P40129 oasc.RecoveryStrategy.run Starting
>> recovery process. core=collection1 recoveringAfterStartup=true
>> [junit4] 2> 89433 T747 C598 P40129 oasc.RecoveryStrategy.doRecovery ######
>> startupVersions=[]
>> [junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish publishing
>> core=collection1 state=recovering collection=collection1
>> [junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish numShards not
>> found on descriptor - reading it from system property
>> [junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 89436 T700 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 89437 T714 oasha.CoreAdminHandler.handleWaitForStateAction
>> Going
>> to wait for coreNodeName: core_node2, state: recovering, checkLive: true,
>> onlyIfLeader: true
>> [junit4] 2> 89437 T700 oasc.Overseer$ClusterStateUpdater.updateState
>> Update
>> state numShards=1 message={
>> [junit4] 2> "operation":"state",
>> [junit4] 2> "state":"recovering",
>> [junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
>> [junit4] 2> "core":"collection1",
>> [junit4] 2> "roles":null,
>> [junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
>> [junit4] 2> "shard":"shard1",
>> [junit4] 2> "collection":"collection1",
>> [junit4] 2> "numShards":"1",
>> [junit4] 2> "core_node_name":"core_node2"}
>> [junit4] 2> 89438 T714 oascc.ZkStateReader.updateClusterState Updating
>> cloud
>> state from ZooKeeper...
>> [junit4] 2> 89439 T714 oasha.CoreAdminHandler.handleWaitForStateAction
>> Will
>> wait a max of 183 seconds to see collection1 (shard1 of collection1) have
>> state: recovering
>> [junit4] 2> 89439 T699 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 89440 T721 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>> [junit4] 2> 89440 T739 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>> [junit4] 2> 89440 T708 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>> [junit4] 2> 89440 T699 oascc.ZkStateReader$2.process A cluster state
>> change:
>> WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>> [junit4] 2> ASYNC NEW_CORE C599 name=collection1
>> org.apache.solr.core.SolrCore@69814a
>> url=http://127.0.0.1:38011/nju/sc/collection1
>> node=127.0.0.1:38011_nju%2Fsc
>> C599_STATE=coll:control_collection core:collection1 props:{state=active,
>> base_url=http://127.0.0.1:38011/nju/sc, core=collection1,
>> node_name=127.0.0.1:38011_nju%2Fsc, leader=true}
>> [junit4] 2> 89447 T690 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-0
>> (1463407845892423680)]}
>> 0 14
>> [junit4] 2> 89448 T694 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-0
>> (1463407845892423681)]}
>> 0 15
>> [junit4] 2> C598_STATE=coll:collection1 core:collection1
>> props:{state=recovering, base_url=http://127.0.0.1:40129/nju/sc,
>> core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
>> [junit4] 2> 89458 T733 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[1-0 (1463407845911298048)]} 0 2
>> [junit4] 2> 89458 T734 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[2-0 (1463407845911298049)]} 0 2
>> [junit4] 2> ASYNC NEW_CORE C600 name=collection1
>> org.apache.solr.core.SolrCore@a6a2f3
>> url=http://127.0.0.1:39738/nju/sc/collection1
>> node=127.0.0.1:39738_nju%2Fsc
>> C600_STATE=coll:collection1 core:collection1 props:{state=active,
>> base_url=http://127.0.0.1:39738/nju/sc, core=collection1,
>> node_name=127.0.0.1:39738_nju%2Fsc, leader=true}
>> [junit4] 2> 89459 T716 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[1-0 (1463407845911298048)]} 0 8
>> [junit4] 2> 89460 T717 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[2-0 (1463407845911298049)]} 0 9
>> [junit4] 2> 89527 T693 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-0
>> (-1463407845989941249)]} 0 1
>> [junit4] 2> 89527 T695 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-0
>> (-1463407845989941248)]} 0 1
>> [junit4] 2> 89532 T730 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {delete=[2-0 (-1463407845994135553)]} 0 0
>> [junit4] 2> 89532 T735 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {delete=[1-0 (-1463407845994135552)]} 0 0
>> [junit4] 2> 89533 T715 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {delete=[2-0 (-1463407845994135553)]} 0 3
>> [junit4] 2> 89533 T716 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {delete=[1-0 (-1463407845994135552)]} 0 3
>> [junit4] 2> 89536 T690 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-1
>> (1463407845999378432)]}
>> 0 1
>> [junit4] 2> 89536 T694 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-1
>> (1463407845999378433)]}
>> 0 1
>> [junit4] 2> 89540 T733 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[2-1 (1463407846002524160)]} 0 1
>> [junit4] 2> 89540 T717 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[2-1 (1463407846002524160)]} 0 2
>> [junit4] 2> 89541 T734 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[1-1 (1463407846002524161)]} 0 1
>> [junit4] 2> 89541 T715 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[1-1 (1463407846002524161)]} 0 3
>> [junit4] 2> 89589 T693 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-2
>> (1463407846054952960)]}
>> 0 1
>> [junit4] 2> 89589 T695 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-2
>> (1463407846054952961)]}
>> 0 1
>> [junit4] 2> 89593 T730 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[2-2 (1463407846058098688)]} 0 0
>> [junit4] 2> 89594 T716 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[2-2 (1463407846058098688)]} 0 3
>> [junit4] 2> 89594 T735 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[1-2 (1463407846058098689)]} 0 1
>> [junit4] 2> 89595 T717 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[1-2 (1463407846058098689)]} 0 4
>> [junit4] 2> 89606 T690 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-3
>> (1463407846072778752)]}
>> 0 1
>> [junit4] 2> 89606 T694 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-3
>> (1463407846073827328)]}
>> 0 0
>> [junit4] 2> 89610 T733 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[2-3 (1463407846075924480)]} 0 0
>> [junit4] 2> 89611 T715 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[2-3 (1463407846075924480)]} 0 3
>> [junit4] 2> 89611 T734 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[1-3 (1463407846075924481)]} 0 1
>> [junit4] 2> 89611 T716 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[1-3 (1463407846075924481)]} 0 3
>> [junit4] 2> 89666 T693 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-4
>> (1463407846135693312)]}
>> 0 1
>> [junit4] 2> 89666 T695 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-4
>> (1463407846135693313)]}
>> 0 1
>> [junit4] 2> 89670 T730 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[2-4 (1463407846138839040)]} 0 1
>> [junit4] 2> 89670 T735 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[1-4 (1463407846138839041)]} 0 0
>> [junit4] 2> 89670 T717 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[2-4 (1463407846138839040)]} 0 2
>> [junit4] 2> 89671 T715 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[1-4 (1463407846138839041)]} 0 3
>> [junit4] 2> 89707 T690 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-5
>> (1463407846179733505)]}
>> 0 0
>> [junit4] 2> 89708 T694 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-5
>> (1463407846179733504)]}
>> 0 1
>> [junit4] 2> 89712 T733 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[2-5 (1463407846181830656)]} 0 1
>> [junit4] 2> 89712 T734 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[1-5 (1463407846182879232)]} 0 1
>> [junit4] 2> 89712 T716 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[2-5 (1463407846181830656)]} 0 3
>> [junit4] 2> 89713 T717 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[1-5 (1463407846182879232)]} 0 3
>> [junit4] 2> 89714 T693 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-2
>> (-1463407846187073536)]} 0 0
>> [junit4] 2> 89715 T695 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-2
>> (-1463407846188122112)]} 0 0
>> [junit4] 2> 89718 T730 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {delete=[2-2 (-1463407846190219264)]} 0 0
>> [junit4] 2> 89718 T735 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {delete=[1-2 (-1463407846190219265)]} 0 0
>> [junit4] 2> 89719 T715 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {delete=[2-2 (-1463407846190219264)]} 0 2
>> [junit4] 2> 89719 T716 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {delete=[1-2 (-1463407846190219265)]} 0 2
>> [junit4] 2> 89721 T690 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-6
>> (1463407846194413568)]}
>> 0 0
>> [junit4] 2> 89721 T694 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-6
>> (1463407846194413569)]}
>> 0 0
>> [junit4] 2> 89725 T733 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[2-6 (1463407846196510720)]} 0 0
>> [junit4] 2> 89725 T734 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[1-6 (1463407846196510721)]} 0 0
>> [junit4] 2> 89726 T717 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[2-6 (1463407846196510720)]} 0 3
>> [junit4] 2> 89726 T715 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[1-6 (1463407846196510721)]} 0 3
>> [junit4] 2> 89742 T693 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-7
>> (1463407846216433664)]}
>> 0 0
>> [junit4] 2> 89743 T695 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-7
>> (1463407846216433665)]}
>> 0 1
>> [junit4] 2> 89746 T730 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[2-7 (1463407846218530816)]} 0 0
>> [junit4] 2> 89747 T735 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[1-7 (1463407846218530817)]} 0 1
>> [junit4] 2> 89747 T716 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[2-7 (1463407846218530816)]} 0 3
>> [junit4] 2> 89747 T717 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[1-7 (1463407846218530817)]} 0 3
>> [junit4] 2> 89794 T690 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-8
>> (1463407846269911040)]}
>> 0 1
>> [junit4] 2> 89794 T694 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-8
>> (1463407846269911041)]}
>> 0 1
>> [junit4] 2> 89798 T733 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[2-8 (1463407846273056768)]} 0 1
>> [junit4] 2> 89798 T734 C598 P40129 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>>
>> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
>> {add=[1-8 (1463407846273056769)]} 0 0
>> [junit4] 2> 89799 T715 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[2-8 (1463407846273056768)]} 0 3
>> [junit4] 2> 89799 T716 C600 P39738 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
>> {add=[1-8 (1463407846273056769)]} 0 3
>> [junit4] 2> 89858 T690 C599 P38011 oasup.LogUpdateProcessor.finish
>> [collection1] webapp=/nju/sc path=/update
>> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-9
>> (1463407846337019904)]}
>> 0 1
>> [junit
>>
>> [...truncated too long message...]
>>
>> asc.Overseer.createOverseerNode ERROR Could not create Overseer node
>> org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
>> [junit4] 2> at
>>
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>> [junit4] 2> at
>>
>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>> [junit4] 2> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>> [junit4] 2>
>> [junit4] 2> 191141 T765 oasc.SolrCore.close [collection1] CLOSING SolrCore
>> org.apache.solr.core.SolrCore@966553
>> [junit4] 2> 191156 T765 oasu.DirectUpdateHandler2.close closing
>> DirectUpdateHandler2{commits=0,autocommits=0,soft
>>
>> autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=86,adds=86,deletesById=48,deletesByQuery=0,errors=0,cumulative_adds=86,cumulative_deletesById=48,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
>> [junit4] 2> 191156 T765 oasu.SolrCoreState.decrefSolrCoreState Closing
>> SolrCoreState
>> [junit4] 2> 191156 T765 oasu.DefaultSolrCoreState.closeIndexWriter
>> SolrCoreState ref count has reached 0 - closing IndexWriter
>> [junit4] 2> 191157 T765 oasu.DefaultSolrCoreState.closeIndexWriter closing
>> IndexWriter with IndexWriterCloser
>> [junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.onCommit
>> SolrDeletionPolicy.onCommit: commits: num=2
>> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6
>>
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_1,generation=1}
>> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6
>>
>> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_2,generation=2}
>> [junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.updateCommits
>> newest commit generation = 2
>> [junit4] 2> 191162 T765 oasc.SolrCore.closeSearcher [collection1] Closing
>> main searcher on request.
>> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing
>> MockDirectoryFactory - 2 directories currently being tracked
>> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue
>> looking
>> to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
>>
>> [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2;done=false>>]
>> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing
>> directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
>> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue
>> looking
>> to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
>>
>> [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index;done=false>>]
>> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing
>> directory:
>> ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
>> [junit4] 2> 191164 T765 oasc.LeaderElector$1.process WARN
>> java.lang.RuntimeException:
>> org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1227)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
>> [junit4] 2> at
>>
>> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
>> [junit4] 2> at
>> org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
>> [junit4] 2> at
>>
>> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>> [junit4] 2> at
>> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>> [junit4] 2> Caused by:
>> org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /overseer
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
>> [junit4] 2> at
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> [junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
>> [junit4] 2> at
>>
>> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
>> [junit4] 2> at
>> org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
>> [junit4] 2> at
>> org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
>> [junit4] 2> ... 14 more
>> [junit4] 2>
>> [junit4] 2> 191164 T765 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 191164 T765 oascc.ZkStateReader$2.process A cluster state
>> change: WatchedEvent state:SyncConnected type:NodeDataChanged
>> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>> [junit4] 2> 191165 T765 oascc.ZkStateReader$2.process WARN ZooKeeper watch
>> triggered, but Solr cannot talk to ZK
>> [junit4] 2> 191165 T765 oasc.DistributedQueue$LatchChildWatcher.process
>> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
>> NodeChildrenChanged
>> [junit4] 2> 191165 T765 oascc.ZkStateReader$3.process WARN ZooKeeper watch
>> triggered, but Solr cannot talk to ZK
>> [junit4] 2> NOTE: test params are: codec=Lucene42,
>> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {},
>> locale=es_GT,
>> timezone=Asia/Brunei
>> [junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51
>> (32-bit)/cpus=8,threads=1,free=80585760,total=235929600
>> [junit4] 2> NOTE: All tests run in this JVM: [DistribCursorPagingTest,
>> ParsingFieldUpdateProcessorsTest, OverseerTest, RecoveryZkTest]
>> [junit4] Completed on J0 in 111.30s, 1 test, 1 failure <<< FAILURES!
>>
>> [...truncated 1081 lines...]
>> BUILD FAILED
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The
>> following
>> error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The
>> following error occurred while executing this line:
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The
>> following error occurred while executing this line:
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490:
>> The following error occurred while executing this line:
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276:
>> The following error occurred while executing this line:
>>
>> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908:
>> There were test failures: 378 suites, 1598 tests, 1 failure, 37 ignored
>> (16
>> assumptions)
>>
>> Total time: 58 minutes 22 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
>> Archiving artifacts
>> Recording test results
>> Email was triggered for: Failure
>> Sending email for trigger: Failure
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build
# 9780 - Still Failing!
Posted by Mark Miller <ma...@gmail.com>.
Yeah, that’s what I was thinking for “Could disable that check for Solr tests that replicate - not very appealing. “
I like all those checks :) I suppose it’s the current best option though, so I’ll do that for now.
--Â
Mark Miller
about.me/markrmiller
On March 23, 2014 at 8:42:34 PM, Michael McCandless (lucene@mikemccandless.com) wrote:
Hmmm, tricky.
Could we conditionalize the check in MDW that throws IOE in openIput
if the file is still open by createOutput? Leave it on by default but
turn it off for these tests?
Mike McCandless
http://blog.mikemccandless.com
On Sun, Mar 23, 2014 at 8:03 PM, Mark Miller <ma...@gmail.com> wrote:
> Issue is that dir.openInput(fileName, IOContext.DEFAULT).close() is not a
> perfect replacement for Directory#fileExists when using the
> MockDirectoryWrapper - if the file is still being written, it throws an
> IOException.
>
> I don't have a solution I'm in love with yet.
>
> Could special case it for MockDirectoryWrapper - not super appealing. Might
> be better if we make some new exceptions though.
>
> Could assume the file exists on IOException or IOException caused by
> RuntimeException - not very appealing.
>
> Could disable that check for Solr tests that replicate - not very appealing.
>
> Could stop using MockDirectoryWrapper on those tests - not very appealing.
>
> --
> Mark Miller
> about.me/markrmiller
>
> On March 23, 2014 at 7:48:47 PM, Mark Miller (markrmiller@gmail.com) wrote:
>
> Okay, looks LUCENE-5543 related. I'll dig into it shortly.
> --
> Mark Miller
> about.me/markrmiller
>
> On March 23, 2014 at 7:44:17 PM, Mark Miller (markrmiller@gmail.com) wrote:
>
> Cause of this seems to be:
>
> 24422 T107 C8 P54679 oasc.SolrException.log ERROR SnapPull failed
> :org.apache.solr.common.SolrException: Index fetch failed :
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:506)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:324)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:446)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:234)
> Caused by: java.io.IOException: MockDirectoryWrapper: file "_0.fdx" is still
> open for writing
> at
> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:568)
> at org.apache.solr.handler.SnapPuller.slowFileExists(SnapPuller.java:789)
> at org.apache.solr.handler.SnapPuller.isIndexStale(SnapPuller.java:805)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:405)
> ... 4 more
> Caused by: java.lang.RuntimeException: unclosed IndexOutput: _0.fdx
> at
> org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:544)
> at
> org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:516)
> at
> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44)
> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:109)
> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120)
> at
> org.apache.lucene.index.StoredFieldsProcessor.initFieldsWriter(StoredFieldsProcessor.java:88)
> at
> org.apache.lucene.index.StoredFieldsProcessor.finishDocument(StoredFieldsProcessor.java:120)
> at
> org.apache.lucene.index.TwoStoredFieldsConsumers.finishDocument(TwoStoredFieldsConsumers.java:65)
> at
> org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:273)
> at
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:271)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:465)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1536)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:236)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:704)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:858)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:557)
> at
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
> at
> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
> at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
> at
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
> at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1957)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:785)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:429)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:134)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:744)
>
> --
> Mark Miller
> about.me/markrmiller
>
> On March 23, 2014 at 7:04:59 PM, Policeman Jenkins Server
> (jenkins@thetaphi.de) wrote:
>
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9780/
> Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
>
> 1 tests failed.
> FAILED: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
>
> Error Message:
> There are still nodes recoverying - waited for 90 seconds
>
> Stack Trace:
> java.lang.AssertionError: There are still nodes recoverying - waited for 90
> seconds
> at __randomizedtesting.SeedInfo.seed([92A83ABF6D073DEE:134EB4A71A585DD2]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:177)
> at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:732)
> at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1495)
> at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:103)
> at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
> at java.lang.Thread.run(Thread.java:744)
>
>
>
>
> Build Log:
> [...truncated 10169 lines...]
> [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
> [junit4] 2> 80886 T677 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl
> (false) and clientAuth (true)
> [junit4] 2> 80887 T677 oas.BaseDistributedSearchTestCase.initHostContext
> Setting hostContext system property: /nju/sc
> [junit4] 2> 80892 T677 oas.SolrTestCaseJ4.setUp ###Starting
> testDistribSearch
> [junit4] 2> Creating dataDir:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1395614468515
> [junit4] 2> 80893 T677 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4] 2> 80894 T678 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting
> server
> [junit4] 2> 80994 T677 oasc.ZkTestServer.run start zk server on port:46640
> [junit4] 2> 80995 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 81204 T684 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@afcd05
> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 81204 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 81205 T677 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 81209 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 81210 T686 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1598a49
> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 81210 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 81211 T677 oascc.SolrZkClient.makePath makePath:
> /collections/collection1
> [junit4] 2> 81213 T677 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/shards
> [junit4] 2> 81215 T677 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection
> [junit4] 2> 81216 T677 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/shards
> [junit4] 2> 81218 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
> to /configs/conf1/solrconfig.xml
> [junit4] 2> 81218 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.xml
> [junit4] 2> 81220 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml
> to /configs/conf1/schema.xml
> [junit4] 2> 81221 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/schema.xml
> [junit4] 2> 81224 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
> to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> [junit4] 2> 81224 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> [junit4] 2> 81226 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
> to /configs/conf1/stopwords.txt
> [junit4] 2> 81226 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/stopwords.txt
> [junit4] 2> 81228 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt
> to /configs/conf1/protwords.txt
> [junit4] 2> 81228 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/protwords.txt
> [junit4] 2> 81230 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml
> to /configs/conf1/currency.xml
> [junit4] 2> 81230 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/currency.xml
> [junit4] 2> 81231 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
> to /configs/conf1/enumsConfig.xml
> [junit4] 2> 81232 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/enumsConfig.xml
> [junit4] 2> 81233 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
> to /configs/conf1/open-exchange-rates.json
> [junit4] 2> 81234 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/open-exchange-rates.json
> [junit4] 2> 81235 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
> to /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4] 2> 81236 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4] 2> 81237 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
> to /configs/conf1/old_synonyms.txt
> [junit4] 2> 81238 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/old_synonyms.txt
> [junit4] 2> 81239 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
> to /configs/conf1/synonyms.txt
> [junit4] 2> 81240 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/synonyms.txt
> [junit4] 2> 81475 T677 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4] 2> 81479 T677 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:38011
> [junit4] 2> 81480 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> [junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not
> configured for solr (NoInitialContextEx)
> [junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome using system
> property solr.solr.home:
> ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864
> [junit4] 2> 81482 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/'
> [junit4] 2> 81536 T677 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/solr.xml
> [junit4] 2> 81599 T677 oasc.CoreContainer.<init> New CoreContainer 27496700
> [junit4] 2> 81599 T677 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/]
> [junit4] 2> 81600 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> socketTimeout to: 90000
> [junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> urlScheme to:
> [junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> connTimeout to: 15000
> [junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxConnectionsPerHost to: 20
> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> corePoolSize to: 0
> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maximumPoolSize to: 2147483647
> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxThreadIdleTime to: 5
> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> sizeOfQueue to: -1
> [junit4] 2> 81603 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> fairnessPolicy to: false
> [junit4] 2> 81603 T677 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 81604 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering
> Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 81604 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
> [junit4] 2> 81604 T677 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:46640/solr
> [junit4] 2> 81605 T677 oasc.ZkController.checkChrootPath zkHost includes
> chroot
> [junit4] 2> 81605 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 81607 T697 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1839970
> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 81607 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 81609 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 81609 T679 oazs.NIOServerCnxn.doIO WARN caught end of stream
> exception EndOfStreamException: Unable to read additional data from client
> sessionid 0x144f11ac9a60002, likely client has closed socket
> [junit4] 2> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> [junit4] 2> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> [junit4] 2> at java.lang.Thread.run(Thread.java:744)
> [junit4] 2>
> [junit4] 2> 81611 T699 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@6336bf
> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 81611 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 81612 T677 oascc.SolrZkClient.makePath makePath: /overseer/queue
> [junit4] 2> 81614 T677 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
> [junit4] 2> 81616 T677 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
> [junit4] 2> 81618 T677 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
> [junit4] 2> 81620 T677 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
> [junit4] 2> 81621 T677 oascc.SolrZkClient.makePath makePath: /live_nodes
> [junit4] 2> 81623 T677 oasc.ZkController.createEphemeralLiveNode Register
> node as live in ZooKeeper:/live_nodes/127.0.0.1:38011_nju%2Fsc
> [junit4] 2> 81625 T677 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:38011_nju%2Fsc
> [junit4] 2> 81627 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect
> [junit4] 2> 81628 T677 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
> [junit4] 2> 81631 T677 oasc.OverseerElectionContext.runLeaderProcess I am
> going to be the leader 127.0.0.1:38011_nju%2Fsc
> [junit4] 2> 81631 T677 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
> [junit4] 2> 81632 T677 oasc.Overseer.start Overseer
> (id=91462989808795651-127.0.0.1:38011_nju%2Fsc-n_0000000000) starting
> [junit4] 2> 81637 T677 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
> [junit4] 2> 81645 T701 oasc.OverseerCollectionProcessor.run Process current
> queue of collection creations
> [junit4] 2> 81645 T677 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
> [junit4] 2> 81646 T701
> oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing
> overseer nodes
> [junit4] 2> 81647 T677 oascc.SolrZkClient.makePath makePath: /aliases.json
> [junit4] 2> 81648 T677
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 81650 T700 oasc.Overseer$ClusterStateUpdater.run Starting to
> work on the main queue
> [junit4] 2> 81655 T702 oasc.ZkController.publish publishing core=collection1
> state=down collection=control_collection
> [junit4] 2> 81656 T702 oasc.ZkController.publish numShards not found on
> descriptor - reading it from system property
> [junit4] 2> 81657 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 81657 T702 oasc.ZkController.waitForCoreNodeName look for our
> core node name
> [junit4] 2> 81658 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":null}
> [junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.createCollection
> Create collection control_collection with shards [shard1]
> [junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState
> Assigning new node to shard shard=shard1
> [junit4] 2> 81661 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 81662 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 82658 T702 oasc.ZkController.waitForShardId waiting to find
> shard id in clusterstate for collection1
> [junit4] 2> 82658 T702 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using instanceDir:
> ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1
> [junit4] 2> 82658 T702 oasc.ZkController.createCollectionZkNode Check for
> collection zkNode:control_collection
> [junit4] 2> 82659 T702 oasc.ZkController.createCollectionZkNode Collection
> zkNode exists
> [junit4] 2> 82659 T702 oascc.ZkStateReader.readConfigName Load collection
> config from:/collections/control_collection
> [junit4] 2> 82660 T702 oascc.ZkStateReader.readConfigName
> path=/collections/control_collection configName=conf1 specified config
> exists in ZooKeeper
> [junit4] 2> 82660 T702 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/'
> [junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/.svn/'
> to classloader
> [junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/classes/'
> to classloader
> [junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/README'
> to classloader
> [junit4] 2> 82714 T702 oasc.SolrConfig.<init> Using Lucene MatchVersion:
> LUCENE_48
> [junit4] 2> 82762 T702 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
> [junit4] 2> 82763 T702 oass.IndexSchema.readSchema Reading Solr Schema from
> schema.xml
> [junit4] 2> 82788 T702 oass.IndexSchema.readSchema [collection1] Schema
> name=test
> [junit4] 2> 83255 T702 oass.IndexSchema.readSchema default search field in
> schema is text
> [junit4] 2> 83258 T702 oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 83260 T702 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 83264 T702 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 83285 T702 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 83286 T702 oasc.SolrCore.<init> [collection1] Opening new
> SolrCore at
> ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/,
> dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/
> [junit4] 2> 83289 T702 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
> [junit4] 2> 83292 T702 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data
> [junit4] 2> 83292 T702 oasc.SolrCore.getNewIndexDir New index directory
> detected: old=null
> new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index/
> [junit4] 2> 83292 T702 oasc.SolrCore.initIndex WARN [collection1] Solr index
> directory
> './org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index'
> doesn't exist. Creating new index...
> [junit4] 2> 83293 T702 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index
> [junit4] 2> 83293 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
> maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
> noCFSRatio=1.0]
> [junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
> [junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 83299 T702 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "nodistrib"
> [junit4] 2> 83310 T702 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "dedupe"
> [junit4] 2> 83313 T702 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "stored_sig"
> [junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "stored_sig"
> [junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> [junit4] 2> 83338 T702 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 83339 T702 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
> [junit4] 2> 83339 T702 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 83341 T702 oasc.RequestHandlers.initHandlersFromConfig created
> standard: solr.StandardRequestHandler
> [junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig created
> /get: solr.RealTimeGetHandler
> [junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig adding
> lazy requestHandler: solr.ReplicationHandler
> [junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created
> /replication: solr.ReplicationHandler
> [junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created
> /update: solr.UpdateRequestHandler
> [junit4] 2> 83344 T702 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4] 2> 83349 T702 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/file: solr.admin.ShowFileRequestHandler
> [junit4] 2> 83351 T702 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 83355 T702 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 83355 T702 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 83357 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogByteSizeMergePolicy:
> [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41,
> maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
> [junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
> [junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 83358 T702 oass.SolrIndexSearcher.<init> Opening
> Searcher@1c23829[collection1] main
> [junit4] 2> 83365 T703 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@1c23829[collection1]
> main{StandardDirectoryReader(segments_1:1:nrt)}
> [junit4] 2> 83368 T702 oasc.CoreContainer.registerCore registering core:
> collection1
> [junit4] 2> 83369 T677 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4] 2> 83369 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
> [junit4] 2> 83369 T706 oasc.ZkController.register Register replica -
> core:collection1 address:http://127.0.0.1:38011/nju/sc
> collection:control_collection shard:shard1
> [junit4] 2> 83370 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 83371 T706 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leader_elect/shard1/election
> [junit4] 2> 83372 T708 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@9657d4
> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 83372 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 83373 T677
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 83375 T677 oasc.ChaosMonkey.monkeyLog monkey: init - expire
> sessions:false cause connection loss:false
> [junit4] 2> 83376 T706 oasc.ShardLeaderElectionContext.runLeaderProcess
> Running the leader process for shard shard1
> [junit4] 2> 83377 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83378 T706
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
> [junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I
> may be the new leader - try and sync
> [junit4] 2> ASYNC NEW_CORE C596 name=collection1
> org.apache.solr.core.SolrCore@69814a
> url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc
> C596_STATE=coll:control_collection core:collection1 props:{state=down,
> base_url=http://127.0.0.1:38011/nju/sc, core=collection1,
> node_name=127.0.0.1:38011_nju%2Fsc}
> [junit4] 2> 83378 T706 C596 P38011 oasc.SyncStrategy.sync Sync replicas to
> http://127.0.0.1:38011/nju/sc/collection1/
> [junit4] 2> 83379 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncReplicas Sync
> Success - now sync replicas to me
> [junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:38011/nju/sc/collection1/ has no replicas
> [junit4] 2> 83379 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I am
> the new leader: http://127.0.0.1:38011/nju/sc/collection1/ shard1
> [junit4] 2> 83380 T706 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leaders/shard1
> [junit4] 2> 83381 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83382 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83382 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83385 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 83386 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83490 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83498 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83539 T706 oasc.ZkController.register We are
> http://127.0.0.1:38011/nju/sc/collection1/ and leader is
> http://127.0.0.1:38011/nju/sc/collection1/
> [junit4] 2> 83539 T706 oasc.ZkController.register No LogReplay needed for
> core=collection1 baseURL=http://127.0.0.1:38011/nju/sc
> [junit4] 2> 83539 T706 oasc.ZkController.checkRecovery I am the leader, no
> recovery necessary
> [junit4] 2> 83540 T706 oasc.ZkController.publish publishing core=collection1
> state=active collection=control_collection
> [junit4] 2> 83540 T706 oasc.ZkController.publish numShards not found on
> descriptor - reading it from system property
> [junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83542 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83542 T706 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 83543 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 83544 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":"core_node1"}
> [junit4] 2> 83546 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83631 T677 oasc.AbstractFullDistribZkTestBase.createJettys
> create jetty 1
> [junit4] 2> 83631 T677 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4] 2> 83651 T677 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:39738
> [junit4] 2> 83652 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> [junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not
> configured for solr (NoInitialContextEx)
> [junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome using system
> property solr.solr.home:
> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998
> [junit4] 2> 83655 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83655 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/'
> [junit4] 2> 83655 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83669 T677 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/solr.xml
> [junit4] 2> 83720 T677 oasc.CoreContainer.<init> New CoreContainer 29042762
> [junit4] 2> 83721 T677 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/]
> [junit4] 2> 83722 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> socketTimeout to: 90000
> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> urlScheme to:
> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> connTimeout to: 15000
> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxConnectionsPerHost to: 20
> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> corePoolSize to: 0
> [junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maximumPoolSize to: 2147483647
> [junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxThreadIdleTime to: 5
> [junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> sizeOfQueue to: -1
> [junit4] 2> 83725 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> fairnessPolicy to: false
> [junit4] 2> 83725 T677 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 83726 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering
> Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 83726 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
> [junit4] 2> 83726 T677 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:46640/solr
> [junit4] 2> 83727 T677 oasc.ZkController.checkChrootPath zkHost includes
> chroot
> [junit4] 2> 83727 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 83729 T719 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@5231ef
> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 83729 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 83731 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 83732 T721 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1075f95
> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 83733 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 83737 T677
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 84742 T677 oasc.ZkController.createEphemeralLiveNode Register
> node as live in ZooKeeper:/live_nodes/127.0.0.1:39738_nju%2Fsc
> [junit4] 2> 84743 T677 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:39738_nju%2Fsc
> [junit4] 2> 84745 T708 oascc.ZkStateReader$3.process Updating live nodes...
> (2)
> [junit4] 2> 84745 T721 oascc.ZkStateReader$3.process Updating live nodes...
> (2)
> [junit4] 2> 84745 T699 oascc.ZkStateReader$3.process Updating live nodes...
> (2)
> [junit4] 2> 84751 T722 oasc.ZkController.publish publishing core=collection1
> state=down collection=collection1
> [junit4] 2> 84752 T722 oasc.ZkController.publish numShards not found on
> descriptor - reading it from system property
> [junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 84755 T722 oasc.ZkController.waitForCoreNodeName look for our
> core node name
> [junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 84756 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 84757 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":null}
> [junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.createCollection
> Create collection collection1 with shards [shard1]
> [junit4] 2> 84758 T700 oasc.Overseer$ClusterStateUpdater.updateState
> Assigning new node to shard shard=shard1
> [junit4] 2> 84759 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 84862 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 84862 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 84862 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 85756 T722 oasc.ZkController.waitForShardId waiting to find
> shard id in clusterstate for collection1
> [junit4] 2> 85756 T722 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using instanceDir:
> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1
> [junit4] 2> 85756 T722 oasc.ZkController.createCollectionZkNode Check for
> collection zkNode:collection1
> [junit4] 2> 85757 T722 oasc.ZkController.createCollectionZkNode Collection
> zkNode exists
> [junit4] 2> 85758 T722 oascc.ZkStateReader.readConfigName Load collection
> config from:/collections/collection1
> [junit4] 2> 85759 T722 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
> [junit4] 2> 85759 T722 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/'
> [junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/.svn/'
> to classloader
> [junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/classes/'
> to classloader
> [junit4] 2> 85761 T722 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/README'
> to classloader
> [junit4] 2> 85807 T722 oasc.SolrConfig.<init> Using Lucene MatchVersion:
> LUCENE_48
> [junit4] 2> 85855 T722 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
> [junit4] 2> 85856 T722 oass.IndexSchema.readSchema Reading Solr Schema from
> schema.xml
> [junit4] 2> 85866 T722 oass.IndexSchema.readSchema [collection1] Schema
> name=test
> [junit4] 2> 86312 T722 oass.IndexSchema.readSchema default search field in
> schema is text
> [junit4] 2> 86315 T722 oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 86317 T722 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 86321 T722 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 86342 T722 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 86343 T722 oasc.SolrCore.<init> [collection1] Opening new
> SolrCore at
> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/,
> dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/
> [junit4] 2> 86343 T722 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
> [junit4] 2> 86346 T722 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1
> [junit4] 2> 86346 T722 oasc.SolrCore.getNewIndexDir New index directory
> detected: old=null
> new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index/
> [junit4] 2> 86346 T722 oasc.SolrCore.initIndex WARN [collection1] Solr index
> directory
> './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index' doesn't
> exist. Creating new index...
> [junit4] 2> 86347 T722 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index
> [junit4] 2> 86347 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
> maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
> noCFSRatio=1.0]
> [junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
> [junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 86353 T722 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "nodistrib"
> [junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "dedupe"
> [junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "stored_sig"
> [junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "stored_sig"
> [junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> [junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
> [junit4] 2> 86356 T722 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 86357 T722 oasc.RequestHandlers.initHandlersFromConfig created
> standard: solr.StandardRequestHandler
> [junit4] 2> 86358 T722 oasc.RequestHandlers.initHandlersFromConfig created
> /get: solr.RealTimeGetHandler
> [junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig adding
> lazy requestHandler: solr.ReplicationHandler
> [junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig created
> /replication: solr.ReplicationHandler
> [junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created
> /update: solr.UpdateRequestHandler
> [junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4] 2> 86361 T722 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/file: solr.admin.ShowFileRequestHandler
> [junit4] 2> 86364 T722 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 86367 T722 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 86367 T722 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 86368 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogByteSizeMergePolicy:
> [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41,
> maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
> [junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
> [junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 86369 T722 oass.SolrIndexSearcher.<init> Opening
> Searcher@383c54[collection1] main
> [junit4] 2> 86376 T723 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@383c54[collection1]
> main{StandardDirectoryReader(segments_1:1:nrt)}
> [junit4] 2> 86381 T722 oasc.CoreContainer.registerCore registering core:
> collection1
> [junit4] 2> 86382 T677 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4] 2> 86382 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
> [junit4] 2> 86382 T726 oasc.ZkController.register Register replica -
> core:collection1 address:http://127.0.0.1:39738/nju/sc
> collection:collection1 shard:shard1
> [junit4] 2> 86383 T726 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard1/election
> [junit4] 2> 86387 T726 oasc.ShardLeaderElectionContext.runLeaderProcess
> Running the leader process for shard shard1
> [junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86389 T726
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
> [junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I
> may be the new leader - try and sync
> [junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> ASYNC NEW_CORE C597 name=collection1
> org.apache.solr.core.SolrCore@a6a2f3
> url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc
> C597_STATE=coll:collection1 core:collection1 props:{state=down,
> base_url=http://127.0.0.1:39738/nju/sc, core=collection1,
> node_name=127.0.0.1:39738_nju%2Fsc}
> [junit4] 2> 86389 T726 C597 P39738 oasc.SyncStrategy.sync Sync replicas to
> http://127.0.0.1:39738/nju/sc/collection1/
> [junit4] 2> 86390 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 86390 T726 C597 P39738 oasc.SyncStrategy.syncReplicas Sync
> Success - now sync replicas to me
> [junit4] 2> 86391 T726 C597 P39738 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:39738/nju/sc/collection1/ has no replicas
> [junit4] 2> 86390 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86391 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I am
> the new leader: http://127.0.0.1:39738/nju/sc/collection1/ shard1
> [junit4] 2> 86391 T726 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard1
> [junit4] 2> 86392 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86394 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86394 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86394 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86397 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 86399 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86501 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86501 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86501 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86547 T726 oasc.ZkController.register We are
> http://127.0.0.1:39738/nju/sc/collection1/ and leader is
> http://127.0.0.1:39738/nju/sc/collection1/
> [junit4] 2> 86547 T726 oasc.ZkController.register No LogReplay needed for
> core=collection1 baseURL=http://127.0.0.1:39738/nju/sc
> [junit4] 2> 86547 T726 oasc.ZkController.checkRecovery I am the leader, no
> recovery necessary
> [junit4] 2> 86548 T726 oasc.ZkController.publish publishing core=collection1
> state=active collection=collection1
> [junit4] 2> 86548 T726 oasc.ZkController.publish numShards not found on
> descriptor - reading it from system property
> [junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86550 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86550 T726 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 86551 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 86551 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":"core_node1"}
> [junit4] 2> 86553 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86655 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86655 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86655 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86667 T677 oasc.AbstractFullDistribZkTestBase.createJettys
> create jetty 2
> [junit4] 2> 86668 T677 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4] 2> 86670 T677 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:40129
> [junit4] 2> 86671 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> [junit4] 2> 86671 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not
> configured for solr (NoInitialContextEx)
> [junit4] 2> 86672 T677 oasc.SolrResourceLoader.locateSolrHome using system
> property solr.solr.home:
> ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005
> [junit4] 2> 86672 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/'
> [junit4] 2> 86704 T677 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/solr.xml
> [junit4] 2> 86751 T677 oasc.CoreContainer.<init> New CoreContainer 10907151
> [junit4] 2> 86751 T677 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/]
> [junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> socketTimeout to: 90000
> [junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> urlScheme to:
> [junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> connTimeout to: 15000
> [junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxConnectionsPerHost to: 20
> [junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> corePoolSize to: 0
> [junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maximumPoolSize to: 2147483647
> [junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxThreadIdleTime to: 5
> [junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> sizeOfQueue to: -1
> [junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> fairnessPolicy to: false
> [junit4] 2> 86756 T677 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 86756 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering
> Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 86756 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
> [junit4] 2> 86757 T677 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:46640/solr
> [junit4] 2> 86757 T677 oasc.ZkController.checkChrootPath zkHost includes
> chroot
> [junit4] 2> 86758 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 86759 T737 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@83c633
> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 86760 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 86762 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 86763 T739 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@18288ea
> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 86763 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 86768 T677
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 87770 T677 oasc.ZkController.createEphemeralLiveNode Register
> node as live in ZooKeeper:/live_nodes/127.0.0.1:40129_nju%2Fsc
> [junit4] 2> 87771 T677 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:40129_nju%2Fsc
> [junit4] 2> 87773 T699 oascc.ZkStateReader$3.process Updating live nodes...
> (3)
> [junit4] 2> 87773 T708 oascc.ZkStateReader$3.process Updating live nodes...
> (3)
> [junit4] 2> 87773 T721 oascc.ZkStateReader$3.process Updating live nodes...
> (3)
> [junit4] 2> 87774 T739 oascc.ZkStateReader$3.process Updating live nodes...
> (3)
> [junit4] 2> 87781 T740 oasc.ZkController.publish publishing core=collection1
> state=down collection=collection1
> [junit4] 2> 87781 T740 oasc.ZkController.publish numShards not found on
> descriptor - reading it from system property
> [junit4] 2> 87783 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 87783 T740 oasc.ZkController.waitForCoreNodeName look for our
> core node name
> [junit4] 2> 87785 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":null}
> [junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState
> Collection already exists with numShards=1
> [junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState
> Assigning new node to shard shard=shard1
> [junit4] 2> 87789 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 87890 T739 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 87890 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 87890 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 87890 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 88785 T740 oasc.ZkController.waitForShardId waiting to find
> shard id in clusterstate for collection1
> [junit4] 2> 88785 T740 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using instanceDir:
> ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1
> [junit4] 2> 88785 T740 oasc.ZkController.createCollectionZkNode Check for
> collection zkNode:collection1
> [junit4] 2> 88786 T740 oasc.ZkController.createCollectionZkNode Collection
> zkNode exists
> [junit4] 2> 88786 T740 oascc.ZkStateReader.readConfigName Load collection
> config from:/collections/collection1
> [junit4] 2> 88787 T740 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
> [junit4] 2> 88787 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/'
> [junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/.svn/'
> to classloader
> [junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/classes/'
> to classloader
> [junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/README'
> to classloader
> [junit4] 2> 88833 T740 oasc.SolrConfig.<init> Using Lucene MatchVersion:
> LUCENE_48
> [junit4] 2> 88882 T740 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
> [junit4] 2> 88883 T740 oass.IndexSchema.readSchema Reading Solr Schema from
> schema.xml
> [junit4] 2> 88895 T740 oass.IndexSchema.readSchema [collection1] Schema
> name=test
> [junit4] 2> 89367 T740 oass.IndexSchema.readSchema default search field in
> schema is text
> [junit4] 2> 89370 T740 oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 89372 T740 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 89375 T740 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 89392 T740 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 89393 T740 oasc.SolrCore.<init> [collection1] Opening new
> SolrCore at
> ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/,
> dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/
> [junit4] 2> 89393 T740 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
> [junit4] 2> 89394 T740 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
> [junit4] 2> 89395 T740 oasc.SolrCore.getNewIndexDir New index directory
> detected: old=null
> new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index/
> [junit4] 2> 89395 T740 oasc.SolrCore.initIndex WARN [collection1] Solr index
> directory
> './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index' doesn't
> exist. Creating new index...
> [junit4] 2> 89395 T740 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
> [junit4] 2> 89396 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
> maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
> noCFSRatio=1.0]
> [junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
> [junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "nodistrib"
> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "dedupe"
> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "stored_sig"
> [junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "stored_sig"
> [junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> [junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 89404 T740 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
> [junit4] 2> 89404 T740 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 89405 T740 oasc.RequestHandlers.initHandlersFromConfig created
> standard: solr.StandardRequestHandler
> [junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created
> /get: solr.RealTimeGetHandler
> [junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig adding
> lazy requestHandler: solr.ReplicationHandler
> [junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created
> /replication: solr.ReplicationHandler
> [junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created
> /update: solr.UpdateRequestHandler
> [junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4] 2> 89408 T740 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/file: solr.admin.ShowFileRequestHandler
> [junit4] 2> 89411 T740 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 89414 T740 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 89414 T740 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 89415 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogByteSizeMergePolicy:
> [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41,
> maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
> [junit4] 2> 89415 T740 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
> [junit4] 2> 89416 T740 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 89416 T740 oass.SolrIndexSearcher.<init> Opening
> Searcher@c599fa[collection1] main
> [junit4] 2> 89422 T741 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@c599fa[collection1]
> main{StandardDirectoryReader(segments_1:1:nrt)}
> [junit4] 2> 89426 T740 oasc.CoreContainer.registerCore registering core:
> collection1
> [junit4] 2> 89426 T677 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4] 2> 89427 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
> [junit4] 2> 89426 T744 oasc.ZkController.register Register replica -
> core:collection1 address:http://127.0.0.1:40129/nju/sc
> collection:collection1 shard:shard1
> [junit4] 2> 89428 T677 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 89430 T744 oasc.ZkController.register We are
> http://127.0.0.1:40129/nju/sc/collection1/ and leader is
> http://127.0.0.1:39738/nju/sc/collection1/
> [junit4] 2> 89430 T744 oasc.ZkController.register No LogReplay needed for
> core=collection1 baseURL=http://127.0.0.1:40129/nju/sc
> [junit4] 2> 89431 T744 oasc.ZkController.checkRecovery Core needs to
> recover:collection1
> [junit4] 2> 89431 T744 oasu.DefaultSolrCoreState.doRecovery Running recovery
> - first canceling any ongoing recovery
> [junit4] 2> 89431 T744 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> ASYNC NEW_CORE C598 name=collection1
> org.apache.solr.core.SolrCore@1ec0a26
> url=http://127.0.0.1:40129/nju/sc/collection1 node=127.0.0.1:40129_nju%2Fsc
> C598_STATE=coll:collection1 core:collection1 props:{state=down,
> base_url=http://127.0.0.1:40129/nju/sc, core=collection1,
> node_name=127.0.0.1:40129_nju%2Fsc}
> [junit4] 2> 89432 T747 C598 P40129 oasc.RecoveryStrategy.run Starting
> recovery process. core=collection1 recoveringAfterStartup=true
> [junit4] 2> 89433 T747 C598 P40129 oasc.RecoveryStrategy.doRecovery ######
> startupVersions=[]
> [junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish publishing
> core=collection1 state=recovering collection=collection1
> [junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
> [junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 89436 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 89437 T714 oasha.CoreAdminHandler.handleWaitForStateAction Going
> to wait for coreNodeName: core_node2, state: recovering, checkLive: true,
> onlyIfLeader: true
> [junit4] 2> 89437 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"recovering",
> [junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":"core_node2"}
> [junit4] 2> 89438 T714 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 89439 T714 oasha.CoreAdminHandler.handleWaitForStateAction Will
> wait a max of 183 seconds to see collection1 (shard1 of collection1) have
> state: recovering
> [junit4] 2> 89439 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 89440 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 89440 T739 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 89440 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 89440 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> ASYNC NEW_CORE C599 name=collection1
> org.apache.solr.core.SolrCore@69814a
> url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc
> C599_STATE=coll:control_collection core:collection1 props:{state=active,
> base_url=http://127.0.0.1:38011/nju/sc, core=collection1,
> node_name=127.0.0.1:38011_nju%2Fsc, leader=true}
> [junit4] 2> 89447 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-0 (1463407845892423680)]}
> 0 14
> [junit4] 2> 89448 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-0 (1463407845892423681)]}
> 0 15
> [junit4] 2> C598_STATE=coll:collection1 core:collection1
> props:{state=recovering, base_url=http://127.0.0.1:40129/nju/sc,
> core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
> [junit4] 2> 89458 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-0 (1463407845911298048)]} 0 2
> [junit4] 2> 89458 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-0 (1463407845911298049)]} 0 2
> [junit4] 2> ASYNC NEW_CORE C600 name=collection1
> org.apache.solr.core.SolrCore@a6a2f3
> url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc
> C600_STATE=coll:collection1 core:collection1 props:{state=active,
> base_url=http://127.0.0.1:39738/nju/sc, core=collection1,
> node_name=127.0.0.1:39738_nju%2Fsc, leader=true}
> [junit4] 2> 89459 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-0 (1463407845911298048)]} 0 8
> [junit4] 2> 89460 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-0 (1463407845911298049)]} 0 9
> [junit4] 2> 89527 T693 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-0
> (-1463407845989941249)]} 0 1
> [junit4] 2> 89527 T695 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-0
> (-1463407845989941248)]} 0 1
> [junit4] 2> 89532 T730 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {delete=[2-0 (-1463407845994135553)]} 0 0
> [junit4] 2> 89532 T735 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {delete=[1-0 (-1463407845994135552)]} 0 0
> [junit4] 2> 89533 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {delete=[2-0 (-1463407845994135553)]} 0 3
> [junit4] 2> 89533 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {delete=[1-0 (-1463407845994135552)]} 0 3
> [junit4] 2> 89536 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-1 (1463407845999378432)]}
> 0 1
> [junit4] 2> 89536 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-1 (1463407845999378433)]}
> 0 1
> [junit4] 2> 89540 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-1 (1463407846002524160)]} 0 1
> [junit4] 2> 89540 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-1 (1463407846002524160)]} 0 2
> [junit4] 2> 89541 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-1 (1463407846002524161)]} 0 1
> [junit4] 2> 89541 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-1 (1463407846002524161)]} 0 3
> [junit4] 2> 89589 T693 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-2 (1463407846054952960)]}
> 0 1
> [junit4] 2> 89589 T695 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-2 (1463407846054952961)]}
> 0 1
> [junit4] 2> 89593 T730 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-2 (1463407846058098688)]} 0 0
> [junit4] 2> 89594 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-2 (1463407846058098688)]} 0 3
> [junit4] 2> 89594 T735 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-2 (1463407846058098689)]} 0 1
> [junit4] 2> 89595 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-2 (1463407846058098689)]} 0 4
> [junit4] 2> 89606 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-3 (1463407846072778752)]}
> 0 1
> [junit4] 2> 89606 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-3 (1463407846073827328)]}
> 0 0
> [junit4] 2> 89610 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-3 (1463407846075924480)]} 0 0
> [junit4] 2> 89611 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-3 (1463407846075924480)]} 0 3
> [junit4] 2> 89611 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-3 (1463407846075924481)]} 0 1
> [junit4] 2> 89611 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-3 (1463407846075924481)]} 0 3
> [junit4] 2> 89666 T693 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-4 (1463407846135693312)]}
> 0 1
> [junit4] 2> 89666 T695 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-4 (1463407846135693313)]}
> 0 1
> [junit4] 2> 89670 T730 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-4 (1463407846138839040)]} 0 1
> [junit4] 2> 89670 T735 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-4 (1463407846138839041)]} 0 0
> [junit4] 2> 89670 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-4 (1463407846138839040)]} 0 2
> [junit4] 2> 89671 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-4 (1463407846138839041)]} 0 3
> [junit4] 2> 89707 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-5 (1463407846179733505)]}
> 0 0
> [junit4] 2> 89708 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-5 (1463407846179733504)]}
> 0 1
> [junit4] 2> 89712 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-5 (1463407846181830656)]} 0 1
> [junit4] 2> 89712 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-5 (1463407846182879232)]} 0 1
> [junit4] 2> 89712 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-5 (1463407846181830656)]} 0 3
> [junit4] 2> 89713 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-5 (1463407846182879232)]} 0 3
> [junit4] 2> 89714 T693 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-2
> (-1463407846187073536)]} 0 0
> [junit4] 2> 89715 T695 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-2
> (-1463407846188122112)]} 0 0
> [junit4] 2> 89718 T730 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {delete=[2-2 (-1463407846190219264)]} 0 0
> [junit4] 2> 89718 T735 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {delete=[1-2 (-1463407846190219265)]} 0 0
> [junit4] 2> 89719 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {delete=[2-2 (-1463407846190219264)]} 0 2
> [junit4] 2> 89719 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {delete=[1-2 (-1463407846190219265)]} 0 2
> [junit4] 2> 89721 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-6 (1463407846194413568)]}
> 0 0
> [junit4] 2> 89721 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-6 (1463407846194413569)]}
> 0 0
> [junit4] 2> 89725 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-6 (1463407846196510720)]} 0 0
> [junit4] 2> 89725 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-6 (1463407846196510721)]} 0 0
> [junit4] 2> 89726 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-6 (1463407846196510720)]} 0 3
> [junit4] 2> 89726 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-6 (1463407846196510721)]} 0 3
> [junit4] 2> 89742 T693 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-7 (1463407846216433664)]}
> 0 0
> [junit4] 2> 89743 T695 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-7 (1463407846216433665)]}
> 0 1
> [junit4] 2> 89746 T730 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-7 (1463407846218530816)]} 0 0
> [junit4] 2> 89747 T735 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-7 (1463407846218530817)]} 0 1
> [junit4] 2> 89747 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-7 (1463407846218530816)]} 0 3
> [junit4] 2> 89747 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-7 (1463407846218530817)]} 0 3
> [junit4] 2> 89794 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-8 (1463407846269911040)]}
> 0 1
> [junit4] 2> 89794 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-8 (1463407846269911041)]}
> 0 1
> [junit4] 2> 89798 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-8 (1463407846273056768)]} 0 1
> [junit4] 2> 89798 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-8 (1463407846273056769)]} 0 0
> [junit4] 2> 89799 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-8 (1463407846273056768)]} 0 3
> [junit4] 2> 89799 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-8 (1463407846273056769)]} 0 3
> [junit4] 2> 89858 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-9 (1463407846337019904)]}
> 0 1
> [junit
>
> [...truncated too long message...]
>
> asc.Overseer.createOverseerNode ERROR Could not create Overseer node
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
> [junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
> [junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> [junit4] 2>
> [junit4] 2> 191141 T765 oasc.SolrCore.close [collection1] CLOSING SolrCore
> org.apache.solr.core.SolrCore@966553
> [junit4] 2> 191156 T765 oasu.DirectUpdateHandler2.close closing
> DirectUpdateHandler2{commits=0,autocommits=0,soft
> autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=86,adds=86,deletesById=48,deletesByQuery=0,errors=0,cumulative_adds=86,cumulative_deletesById=48,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
> [junit4] 2> 191156 T765 oasu.SolrCoreState.decrefSolrCoreState Closing
> SolrCoreState
> [junit4] 2> 191156 T765 oasu.DefaultSolrCoreState.closeIndexWriter
> SolrCoreState ref count has reached 0 - closing IndexWriter
> [junit4] 2> 191157 T765 oasu.DefaultSolrCoreState.closeIndexWriter closing
> IndexWriter with IndexWriterCloser
> [junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=2
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_1,generation=1}
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_2,generation=2}
> [junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 2
> [junit4] 2> 191162 T765 oasc.SolrCore.closeSearcher [collection1] Closing
> main searcher on request.
> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing
> MockDirectoryFactory - 2 directories currently being tracked
> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking
> to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
> [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2;done=false>>]
> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing
> directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking
> to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
> [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index;done=false>>]
> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing
> directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
> [junit4] 2> 191164 T765 oasc.LeaderElector$1.process WARN
> java.lang.RuntimeException:
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer
> [junit4] 2> at
> org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1227)
> [junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
> [junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> [junit4] 2> Caused by:
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
> [junit4] 2> ... 14 more
> [junit4] 2>
> [junit4] 2> 191164 T765 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 191164 T765 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 191165 T765 oascc.ZkStateReader$2.process WARN ZooKeeper watch
> triggered, but Solr cannot talk to ZK
> [junit4] 2> 191165 T765 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 191165 T765 oascc.ZkStateReader$3.process WARN ZooKeeper watch
> triggered, but Solr cannot talk to ZK
> [junit4] 2> NOTE: test params are: codec=Lucene42,
> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_GT,
> timezone=Asia/Brunei
> [junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51
> (32-bit)/cpus=8,threads=1,free=80585760,total=235929600
> [junit4] 2> NOTE: All tests run in this JVM: [DistribCursorPagingTest,
> ParsingFieldUpdateProcessorsTest, OverseerTest, RecoveryZkTest]
> [junit4] Completed on J0 in 111.30s, 1 test, 1 failure <<< FAILURES!
>
> [...truncated 1081 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following
> error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908:
> There were test failures: 378 suites, 1598 tests, 1 failure, 37 ignored (16
> assumptions)
>
> Total time: 58 minutes 22 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build #
9780 - Still Failing!
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hmmm, tricky.
Could we conditionalize the check in MDW that throws IOE in openIput
if the file is still open by createOutput? Leave it on by default but
turn it off for these tests?
Mike McCandless
http://blog.mikemccandless.com
On Sun, Mar 23, 2014 at 8:03 PM, Mark Miller <ma...@gmail.com> wrote:
> Issue is that dir.openInput(fileName, IOContext.DEFAULT).close() is not a
> perfect replacement for Directory#fileExists when using the
> MockDirectoryWrapper - if the file is still being written, it throws an
> IOException.
>
> I don't have a solution I'm in love with yet.
>
> Could special case it for MockDirectoryWrapper - not super appealing. Might
> be better if we make some new exceptions though.
>
> Could assume the file exists on IOException or IOException caused by
> RuntimeException - not very appealing.
>
> Could disable that check for Solr tests that replicate - not very appealing.
>
> Could stop using MockDirectoryWrapper on those tests - not very appealing.
>
> --
> Mark Miller
> about.me/markrmiller
>
> On March 23, 2014 at 7:48:47 PM, Mark Miller (markrmiller@gmail.com) wrote:
>
> Okay, looks LUCENE-5543 related. I'll dig into it shortly.
> --
> Mark Miller
> about.me/markrmiller
>
> On March 23, 2014 at 7:44:17 PM, Mark Miller (markrmiller@gmail.com) wrote:
>
> Cause of this seems to be:
>
> 24422 T107 C8 P54679 oasc.SolrException.log ERROR SnapPull failed
> :org.apache.solr.common.SolrException: Index fetch failed :
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:506)
> at
> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:324)
> at
> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
> at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:446)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:234)
> Caused by: java.io.IOException: MockDirectoryWrapper: file "_0.fdx" is still
> open for writing
> at
> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:568)
> at org.apache.solr.handler.SnapPuller.slowFileExists(SnapPuller.java:789)
> at org.apache.solr.handler.SnapPuller.isIndexStale(SnapPuller.java:805)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:405)
> ... 4 more
> Caused by: java.lang.RuntimeException: unclosed IndexOutput: _0.fdx
> at
> org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:544)
> at
> org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:516)
> at
> org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44)
> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:109)
> at
> org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120)
> at
> org.apache.lucene.index.StoredFieldsProcessor.initFieldsWriter(StoredFieldsProcessor.java:88)
> at
> org.apache.lucene.index.StoredFieldsProcessor.finishDocument(StoredFieldsProcessor.java:120)
> at
> org.apache.lucene.index.TwoStoredFieldsConsumers.finishDocument(TwoStoredFieldsConsumers.java:65)
> at
> org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:273)
> at
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:271)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:465)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1536)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:236)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:704)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:858)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:557)
> at
> org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
> at
> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
> at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
> at
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
> at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1957)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:785)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:429)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:134)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> at org.eclipse.jetty.server.Server.handle(Server.java:368)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> at
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> at
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
> at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> at
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
> at
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> at java.lang.Thread.run(Thread.java:744)
>
> --
> Mark Miller
> about.me/markrmiller
>
> On March 23, 2014 at 7:04:59 PM, Policeman Jenkins Server
> (jenkins@thetaphi.de) wrote:
>
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9780/
> Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
>
> 1 tests failed.
> FAILED: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
>
> Error Message:
> There are still nodes recoverying - waited for 90 seconds
>
> Stack Trace:
> java.lang.AssertionError: There are still nodes recoverying - waited for 90
> seconds
> at __randomizedtesting.SeedInfo.seed([92A83ABF6D073DEE:134EB4A71A585DD2]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:177)
> at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:732)
> at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1495)
> at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:103)
> at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
> at java.lang.Thread.run(Thread.java:744)
>
>
>
>
> Build Log:
> [...truncated 10169 lines...]
> [junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
> [junit4] 2> 80886 T677 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl
> (false) and clientAuth (true)
> [junit4] 2> 80887 T677 oas.BaseDistributedSearchTestCase.initHostContext
> Setting hostContext system property: /nju/sc
> [junit4] 2> 80892 T677 oas.SolrTestCaseJ4.setUp ###Starting
> testDistribSearch
> [junit4] 2> Creating dataDir:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1395614468515
> [junit4] 2> 80893 T677 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4] 2> 80894 T678 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting
> server
> [junit4] 2> 80994 T677 oasc.ZkTestServer.run start zk server on port:46640
> [junit4] 2> 80995 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 81204 T684 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@afcd05
> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 81204 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 81205 T677 oascc.SolrZkClient.makePath makePath: /solr
> [junit4] 2> 81209 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 81210 T686 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1598a49
> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 81210 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 81211 T677 oascc.SolrZkClient.makePath makePath:
> /collections/collection1
> [junit4] 2> 81213 T677 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/shards
> [junit4] 2> 81215 T677 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection
> [junit4] 2> 81216 T677 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/shards
> [junit4] 2> 81218 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
> to /configs/conf1/solrconfig.xml
> [junit4] 2> 81218 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.xml
> [junit4] 2> 81220 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml
> to /configs/conf1/schema.xml
> [junit4] 2> 81221 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/schema.xml
> [junit4] 2> 81224 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
> to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> [junit4] 2> 81224 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
> [junit4] 2> 81226 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
> to /configs/conf1/stopwords.txt
> [junit4] 2> 81226 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/stopwords.txt
> [junit4] 2> 81228 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt
> to /configs/conf1/protwords.txt
> [junit4] 2> 81228 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/protwords.txt
> [junit4] 2> 81230 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml
> to /configs/conf1/currency.xml
> [junit4] 2> 81230 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/currency.xml
> [junit4] 2> 81231 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
> to /configs/conf1/enumsConfig.xml
> [junit4] 2> 81232 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/enumsConfig.xml
> [junit4] 2> 81233 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
> to /configs/conf1/open-exchange-rates.json
> [junit4] 2> 81234 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/open-exchange-rates.json
> [junit4] 2> 81235 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
> to /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4] 2> 81236 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4] 2> 81237 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
> to /configs/conf1/old_synonyms.txt
> [junit4] 2> 81238 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/old_synonyms.txt
> [junit4] 2> 81239 T677 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
> to /configs/conf1/synonyms.txt
> [junit4] 2> 81240 T677 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/synonyms.txt
> [junit4] 2> 81475 T677 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4] 2> 81479 T677 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:38011
> [junit4] 2> 81480 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> [junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not
> configured for solr (NoInitialContextEx)
> [junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome using system
> property solr.solr.home:
> ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864
> [junit4] 2> 81482 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/'
> [junit4] 2> 81536 T677 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/solr.xml
> [junit4] 2> 81599 T677 oasc.CoreContainer.<init> New CoreContainer 27496700
> [junit4] 2> 81599 T677 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/]
> [junit4] 2> 81600 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> socketTimeout to: 90000
> [junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> urlScheme to:
> [junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> connTimeout to: 15000
> [junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxConnectionsPerHost to: 20
> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> corePoolSize to: 0
> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maximumPoolSize to: 2147483647
> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxThreadIdleTime to: 5
> [junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> sizeOfQueue to: -1
> [junit4] 2> 81603 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> fairnessPolicy to: false
> [junit4] 2> 81603 T677 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 81604 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering
> Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 81604 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
> [junit4] 2> 81604 T677 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:46640/solr
> [junit4] 2> 81605 T677 oasc.ZkController.checkChrootPath zkHost includes
> chroot
> [junit4] 2> 81605 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 81607 T697 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1839970
> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 81607 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 81609 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 81609 T679 oazs.NIOServerCnxn.doIO WARN caught end of stream
> exception EndOfStreamException: Unable to read additional data from client
> sessionid 0x144f11ac9a60002, likely client has closed socket
> [junit4] 2> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
> [junit4] 2> at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> [junit4] 2> at java.lang.Thread.run(Thread.java:744)
> [junit4] 2>
> [junit4] 2> 81611 T699 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@6336bf
> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 81611 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 81612 T677 oascc.SolrZkClient.makePath makePath: /overseer/queue
> [junit4] 2> 81614 T677 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
> [junit4] 2> 81616 T677 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
> [junit4] 2> 81618 T677 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
> [junit4] 2> 81620 T677 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
> [junit4] 2> 81621 T677 oascc.SolrZkClient.makePath makePath: /live_nodes
> [junit4] 2> 81623 T677 oasc.ZkController.createEphemeralLiveNode Register
> node as live in ZooKeeper:/live_nodes/127.0.0.1:38011_nju%2Fsc
> [junit4] 2> 81625 T677 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:38011_nju%2Fsc
> [junit4] 2> 81627 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect
> [junit4] 2> 81628 T677 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
> [junit4] 2> 81631 T677 oasc.OverseerElectionContext.runLeaderProcess I am
> going to be the leader 127.0.0.1:38011_nju%2Fsc
> [junit4] 2> 81631 T677 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
> [junit4] 2> 81632 T677 oasc.Overseer.start Overseer
> (id=91462989808795651-127.0.0.1:38011_nju%2Fsc-n_0000000000) starting
> [junit4] 2> 81637 T677 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
> [junit4] 2> 81645 T701 oasc.OverseerCollectionProcessor.run Process current
> queue of collection creations
> [junit4] 2> 81645 T677 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
> [junit4] 2> 81646 T701
> oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing
> overseer nodes
> [junit4] 2> 81647 T677 oascc.SolrZkClient.makePath makePath: /aliases.json
> [junit4] 2> 81648 T677
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 81650 T700 oasc.Overseer$ClusterStateUpdater.run Starting to
> work on the main queue
> [junit4] 2> 81655 T702 oasc.ZkController.publish publishing core=collection1
> state=down collection=control_collection
> [junit4] 2> 81656 T702 oasc.ZkController.publish numShards not found on
> descriptor - reading it from system property
> [junit4] 2> 81657 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 81657 T702 oasc.ZkController.waitForCoreNodeName look for our
> core node name
> [junit4] 2> 81658 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":null}
> [junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.createCollection
> Create collection control_collection with shards [shard1]
> [junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState
> Assigning new node to shard shard=shard1
> [junit4] 2> 81661 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 81662 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 82658 T702 oasc.ZkController.waitForShardId waiting to find
> shard id in clusterstate for collection1
> [junit4] 2> 82658 T702 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using instanceDir:
> ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1
> [junit4] 2> 82658 T702 oasc.ZkController.createCollectionZkNode Check for
> collection zkNode:control_collection
> [junit4] 2> 82659 T702 oasc.ZkController.createCollectionZkNode Collection
> zkNode exists
> [junit4] 2> 82659 T702 oascc.ZkStateReader.readConfigName Load collection
> config from:/collections/control_collection
> [junit4] 2> 82660 T702 oascc.ZkStateReader.readConfigName
> path=/collections/control_collection configName=conf1 specified config
> exists in ZooKeeper
> [junit4] 2> 82660 T702 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/'
> [junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/.svn/'
> to classloader
> [junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/classes/'
> to classloader
> [junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/README'
> to classloader
> [junit4] 2> 82714 T702 oasc.SolrConfig.<init> Using Lucene MatchVersion:
> LUCENE_48
> [junit4] 2> 82762 T702 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
> [junit4] 2> 82763 T702 oass.IndexSchema.readSchema Reading Solr Schema from
> schema.xml
> [junit4] 2> 82788 T702 oass.IndexSchema.readSchema [collection1] Schema
> name=test
> [junit4] 2> 83255 T702 oass.IndexSchema.readSchema default search field in
> schema is text
> [junit4] 2> 83258 T702 oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 83260 T702 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 83264 T702 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 83285 T702 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 83286 T702 oasc.SolrCore.<init> [collection1] Opening new
> SolrCore at
> ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/,
> dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/
> [junit4] 2> 83289 T702 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
> [junit4] 2> 83292 T702 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data
> [junit4] 2> 83292 T702 oasc.SolrCore.getNewIndexDir New index directory
> detected: old=null
> new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index/
> [junit4] 2> 83292 T702 oasc.SolrCore.initIndex WARN [collection1] Solr index
> directory
> './org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index'
> doesn't exist. Creating new index...
> [junit4] 2> 83293 T702 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index
> [junit4] 2> 83293 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
> maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
> noCFSRatio=1.0]
> [junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
> [junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 83299 T702 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "nodistrib"
> [junit4] 2> 83310 T702 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "dedupe"
> [junit4] 2> 83313 T702 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "stored_sig"
> [junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "stored_sig"
> [junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> [junit4] 2> 83338 T702 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 83339 T702 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
> [junit4] 2> 83339 T702 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 83341 T702 oasc.RequestHandlers.initHandlersFromConfig created
> standard: solr.StandardRequestHandler
> [junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig created
> /get: solr.RealTimeGetHandler
> [junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig adding
> lazy requestHandler: solr.ReplicationHandler
> [junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created
> /replication: solr.ReplicationHandler
> [junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created
> /update: solr.UpdateRequestHandler
> [junit4] 2> 83344 T702 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4] 2> 83349 T702 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/file: solr.admin.ShowFileRequestHandler
> [junit4] 2> 83351 T702 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 83355 T702 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 83355 T702 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 83357 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogByteSizeMergePolicy:
> [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41,
> maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
> [junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
> [junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 83358 T702 oass.SolrIndexSearcher.<init> Opening
> Searcher@1c23829[collection1] main
> [junit4] 2> 83365 T703 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@1c23829[collection1]
> main{StandardDirectoryReader(segments_1:1:nrt)}
> [junit4] 2> 83368 T702 oasc.CoreContainer.registerCore registering core:
> collection1
> [junit4] 2> 83369 T677 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4] 2> 83369 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
> [junit4] 2> 83369 T706 oasc.ZkController.register Register replica -
> core:collection1 address:http://127.0.0.1:38011/nju/sc
> collection:control_collection shard:shard1
> [junit4] 2> 83370 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 83371 T706 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leader_elect/shard1/election
> [junit4] 2> 83372 T708 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@9657d4
> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 83372 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 83373 T677
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 83375 T677 oasc.ChaosMonkey.monkeyLog monkey: init - expire
> sessions:false cause connection loss:false
> [junit4] 2> 83376 T706 oasc.ShardLeaderElectionContext.runLeaderProcess
> Running the leader process for shard shard1
> [junit4] 2> 83377 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83378 T706
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
> [junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I
> may be the new leader - try and sync
> [junit4] 2> ASYNC NEW_CORE C596 name=collection1
> org.apache.solr.core.SolrCore@69814a
> url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc
> C596_STATE=coll:control_collection core:collection1 props:{state=down,
> base_url=http://127.0.0.1:38011/nju/sc, core=collection1,
> node_name=127.0.0.1:38011_nju%2Fsc}
> [junit4] 2> 83378 T706 C596 P38011 oasc.SyncStrategy.sync Sync replicas to
> http://127.0.0.1:38011/nju/sc/collection1/
> [junit4] 2> 83379 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncReplicas Sync
> Success - now sync replicas to me
> [junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:38011/nju/sc/collection1/ has no replicas
> [junit4] 2> 83379 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I am
> the new leader: http://127.0.0.1:38011/nju/sc/collection1/ shard1
> [junit4] 2> 83380 T706 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leaders/shard1
> [junit4] 2> 83381 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83382 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83382 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83385 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 83386 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83490 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83498 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83539 T706 oasc.ZkController.register We are
> http://127.0.0.1:38011/nju/sc/collection1/ and leader is
> http://127.0.0.1:38011/nju/sc/collection1/
> [junit4] 2> 83539 T706 oasc.ZkController.register No LogReplay needed for
> core=collection1 baseURL=http://127.0.0.1:38011/nju/sc
> [junit4] 2> 83539 T706 oasc.ZkController.checkRecovery I am the leader, no
> recovery necessary
> [junit4] 2> 83540 T706 oasc.ZkController.publish publishing core=collection1
> state=active collection=control_collection
> [junit4] 2> 83540 T706 oasc.ZkController.publish numShards not found on
> descriptor - reading it from system property
> [junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83542 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83542 T706 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 83543 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 83544 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"control_collection",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":"core_node1"}
> [junit4] 2> 83546 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 83631 T677 oasc.AbstractFullDistribZkTestBase.createJettys
> create jetty 1
> [junit4] 2> 83631 T677 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4] 2> 83651 T677 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:39738
> [junit4] 2> 83652 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> [junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not
> configured for solr (NoInitialContextEx)
> [junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome using system
> property solr.solr.home:
> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998
> [junit4] 2> 83655 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83655 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/'
> [junit4] 2> 83655 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 83669 T677 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/solr.xml
> [junit4] 2> 83720 T677 oasc.CoreContainer.<init> New CoreContainer 29042762
> [junit4] 2> 83721 T677 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/]
> [junit4] 2> 83722 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> socketTimeout to: 90000
> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> urlScheme to:
> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> connTimeout to: 15000
> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxConnectionsPerHost to: 20
> [junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> corePoolSize to: 0
> [junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maximumPoolSize to: 2147483647
> [junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxThreadIdleTime to: 5
> [junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> sizeOfQueue to: -1
> [junit4] 2> 83725 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> fairnessPolicy to: false
> [junit4] 2> 83725 T677 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 83726 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering
> Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 83726 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
> [junit4] 2> 83726 T677 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:46640/solr
> [junit4] 2> 83727 T677 oasc.ZkController.checkChrootPath zkHost includes
> chroot
> [junit4] 2> 83727 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 83729 T719 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@5231ef
> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 83729 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 83731 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 83732 T721 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1075f95
> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 83733 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 83737 T677
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 84742 T677 oasc.ZkController.createEphemeralLiveNode Register
> node as live in ZooKeeper:/live_nodes/127.0.0.1:39738_nju%2Fsc
> [junit4] 2> 84743 T677 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:39738_nju%2Fsc
> [junit4] 2> 84745 T708 oascc.ZkStateReader$3.process Updating live nodes...
> (2)
> [junit4] 2> 84745 T721 oascc.ZkStateReader$3.process Updating live nodes...
> (2)
> [junit4] 2> 84745 T699 oascc.ZkStateReader$3.process Updating live nodes...
> (2)
> [junit4] 2> 84751 T722 oasc.ZkController.publish publishing core=collection1
> state=down collection=collection1
> [junit4] 2> 84752 T722 oasc.ZkController.publish numShards not found on
> descriptor - reading it from system property
> [junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 84755 T722 oasc.ZkController.waitForCoreNodeName look for our
> core node name
> [junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 84756 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 84757 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":null}
> [junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.createCollection
> Create collection collection1 with shards [shard1]
> [junit4] 2> 84758 T700 oasc.Overseer$ClusterStateUpdater.updateState
> Assigning new node to shard shard=shard1
> [junit4] 2> 84759 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 84862 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 84862 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 84862 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 85756 T722 oasc.ZkController.waitForShardId waiting to find
> shard id in clusterstate for collection1
> [junit4] 2> 85756 T722 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using instanceDir:
> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1
> [junit4] 2> 85756 T722 oasc.ZkController.createCollectionZkNode Check for
> collection zkNode:collection1
> [junit4] 2> 85757 T722 oasc.ZkController.createCollectionZkNode Collection
> zkNode exists
> [junit4] 2> 85758 T722 oascc.ZkStateReader.readConfigName Load collection
> config from:/collections/collection1
> [junit4] 2> 85759 T722 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
> [junit4] 2> 85759 T722 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/'
> [junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/.svn/'
> to classloader
> [junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/classes/'
> to classloader
> [junit4] 2> 85761 T722 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/README'
> to classloader
> [junit4] 2> 85807 T722 oasc.SolrConfig.<init> Using Lucene MatchVersion:
> LUCENE_48
> [junit4] 2> 85855 T722 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
> [junit4] 2> 85856 T722 oass.IndexSchema.readSchema Reading Solr Schema from
> schema.xml
> [junit4] 2> 85866 T722 oass.IndexSchema.readSchema [collection1] Schema
> name=test
> [junit4] 2> 86312 T722 oass.IndexSchema.readSchema default search field in
> schema is text
> [junit4] 2> 86315 T722 oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 86317 T722 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 86321 T722 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 86342 T722 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 86343 T722 oasc.SolrCore.<init> [collection1] Opening new
> SolrCore at
> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/,
> dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/
> [junit4] 2> 86343 T722 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
> [junit4] 2> 86346 T722 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1
> [junit4] 2> 86346 T722 oasc.SolrCore.getNewIndexDir New index directory
> detected: old=null
> new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index/
> [junit4] 2> 86346 T722 oasc.SolrCore.initIndex WARN [collection1] Solr index
> directory
> './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index' doesn't
> exist. Creating new index...
> [junit4] 2> 86347 T722 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index
> [junit4] 2> 86347 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
> maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
> noCFSRatio=1.0]
> [junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
> [junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 86353 T722 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "nodistrib"
> [junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "dedupe"
> [junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "stored_sig"
> [junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "stored_sig"
> [junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> [junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
> [junit4] 2> 86356 T722 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 86357 T722 oasc.RequestHandlers.initHandlersFromConfig created
> standard: solr.StandardRequestHandler
> [junit4] 2> 86358 T722 oasc.RequestHandlers.initHandlersFromConfig created
> /get: solr.RealTimeGetHandler
> [junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig adding
> lazy requestHandler: solr.ReplicationHandler
> [junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig created
> /replication: solr.ReplicationHandler
> [junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created
> /update: solr.UpdateRequestHandler
> [junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4] 2> 86361 T722 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/file: solr.admin.ShowFileRequestHandler
> [junit4] 2> 86364 T722 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 86367 T722 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 86367 T722 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 86368 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogByteSizeMergePolicy:
> [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41,
> maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
> [junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
> [junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 86369 T722 oass.SolrIndexSearcher.<init> Opening
> Searcher@383c54[collection1] main
> [junit4] 2> 86376 T723 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@383c54[collection1]
> main{StandardDirectoryReader(segments_1:1:nrt)}
> [junit4] 2> 86381 T722 oasc.CoreContainer.registerCore registering core:
> collection1
> [junit4] 2> 86382 T677 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4] 2> 86382 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
> [junit4] 2> 86382 T726 oasc.ZkController.register Register replica -
> core:collection1 address:http://127.0.0.1:39738/nju/sc
> collection:collection1 shard:shard1
> [junit4] 2> 86383 T726 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard1/election
> [junit4] 2> 86387 T726 oasc.ShardLeaderElectionContext.runLeaderProcess
> Running the leader process for shard shard1
> [junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86389 T726
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
> [junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I
> may be the new leader - try and sync
> [junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> ASYNC NEW_CORE C597 name=collection1
> org.apache.solr.core.SolrCore@a6a2f3
> url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc
> C597_STATE=coll:collection1 core:collection1 props:{state=down,
> base_url=http://127.0.0.1:39738/nju/sc, core=collection1,
> node_name=127.0.0.1:39738_nju%2Fsc}
> [junit4] 2> 86389 T726 C597 P39738 oasc.SyncStrategy.sync Sync replicas to
> http://127.0.0.1:39738/nju/sc/collection1/
> [junit4] 2> 86390 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 86390 T726 C597 P39738 oasc.SyncStrategy.syncReplicas Sync
> Success - now sync replicas to me
> [junit4] 2> 86391 T726 C597 P39738 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:39738/nju/sc/collection1/ has no replicas
> [junit4] 2> 86390 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86391 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I am
> the new leader: http://127.0.0.1:39738/nju/sc/collection1/ shard1
> [junit4] 2> 86391 T726 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard1
> [junit4] 2> 86392 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86394 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86394 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86394 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86397 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 86399 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86501 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86501 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86501 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86547 T726 oasc.ZkController.register We are
> http://127.0.0.1:39738/nju/sc/collection1/ and leader is
> http://127.0.0.1:39738/nju/sc/collection1/
> [junit4] 2> 86547 T726 oasc.ZkController.register No LogReplay needed for
> core=collection1 baseURL=http://127.0.0.1:39738/nju/sc
> [junit4] 2> 86547 T726 oasc.ZkController.checkRecovery I am the leader, no
> recovery necessary
> [junit4] 2> 86548 T726 oasc.ZkController.publish publishing core=collection1
> state=active collection=collection1
> [junit4] 2> 86548 T726 oasc.ZkController.publish numShards not found on
> descriptor - reading it from system property
> [junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86550 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86550 T726 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 86551 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 86551 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"active",
> [junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":"core_node1"}
> [junit4] 2> 86553 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 86655 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86655 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86655 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4] 2> 86667 T677 oasc.AbstractFullDistribZkTestBase.createJettys
> create jetty 2
> [junit4] 2> 86668 T677 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4] 2> 86670 T677 oejs.AbstractConnector.doStart Started
> SelectChannelConnector@127.0.0.1:40129
> [junit4] 2> 86671 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
> [junit4] 2> 86671 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not
> configured for solr (NoInitialContextEx)
> [junit4] 2> 86672 T677 oasc.SolrResourceLoader.locateSolrHome using system
> property solr.solr.home:
> ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005
> [junit4] 2> 86672 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/'
> [junit4] 2> 86704 T677 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/solr.xml
> [junit4] 2> 86751 T677 oasc.CoreContainer.<init> New CoreContainer 10907151
> [junit4] 2> 86751 T677 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/]
> [junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> socketTimeout to: 90000
> [junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> urlScheme to:
> [junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> connTimeout to: 15000
> [junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxConnectionsPerHost to: 20
> [junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> corePoolSize to: 0
> [junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maximumPoolSize to: 2147483647
> [junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> maxThreadIdleTime to: 5
> [junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> sizeOfQueue to: -1
> [junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting
> fairnessPolicy to: false
> [junit4] 2> 86756 T677 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
> [junit4] 2> 86756 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering
> Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
> [junit4] 2> 86756 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
> [junit4] 2> 86757 T677 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:46640/solr
> [junit4] 2> 86757 T677 oasc.ZkController.checkChrootPath zkHost includes
> chroot
> [junit4] 2> 86758 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 86759 T737 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@83c633
> name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 86760 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 86762 T677 oascc.ConnectionManager.waitForConnected Waiting for
> client to connect to ZooKeeper
> [junit4] 2> 86763 T739 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@18288ea
> name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
> [junit4] 2> 86763 T677 oascc.ConnectionManager.waitForConnected Client is
> connected to ZooKeeper
> [junit4] 2> 86768 T677
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
> [junit4] 2> 87770 T677 oasc.ZkController.createEphemeralLiveNode Register
> node as live in ZooKeeper:/live_nodes/127.0.0.1:40129_nju%2Fsc
> [junit4] 2> 87771 T677 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:40129_nju%2Fsc
> [junit4] 2> 87773 T699 oascc.ZkStateReader$3.process Updating live nodes...
> (3)
> [junit4] 2> 87773 T708 oascc.ZkStateReader$3.process Updating live nodes...
> (3)
> [junit4] 2> 87773 T721 oascc.ZkStateReader$3.process Updating live nodes...
> (3)
> [junit4] 2> 87774 T739 oascc.ZkStateReader$3.process Updating live nodes...
> (3)
> [junit4] 2> 87781 T740 oasc.ZkController.publish publishing core=collection1
> state=down collection=collection1
> [junit4] 2> 87781 T740 oasc.ZkController.publish numShards not found on
> descriptor - reading it from system property
> [junit4] 2> 87783 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 87783 T740 oasc.ZkController.waitForCoreNodeName look for our
> core node name
> [junit4] 2> 87785 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
> [junit4] 2> "shard":null,
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":null}
> [junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState
> Collection already exists with numShards=1
> [junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState
> Assigning new node to shard shard=shard1
> [junit4] 2> 87789 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 87890 T739 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 87890 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 87890 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 87890 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 88785 T740 oasc.ZkController.waitForShardId waiting to find
> shard id in clusterstate for collection1
> [junit4] 2> 88785 T740 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using instanceDir:
> ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1
> [junit4] 2> 88785 T740 oasc.ZkController.createCollectionZkNode Check for
> collection zkNode:collection1
> [junit4] 2> 88786 T740 oasc.ZkController.createCollectionZkNode Collection
> zkNode exists
> [junit4] 2> 88786 T740 oascc.ZkStateReader.readConfigName Load collection
> config from:/collections/collection1
> [junit4] 2> 88787 T740 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
> [junit4] 2> 88787 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader
> for directory:
> './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/'
> [junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/.svn/'
> to classloader
> [junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/classes/'
> to classloader
> [junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/README'
> to classloader
> [junit4] 2> 88833 T740 oasc.SolrConfig.<init> Using Lucene MatchVersion:
> LUCENE_48
> [junit4] 2> 88882 T740 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
> [junit4] 2> 88883 T740 oass.IndexSchema.readSchema Reading Solr Schema from
> schema.xml
> [junit4] 2> 88895 T740 oass.IndexSchema.readSchema [collection1] Schema
> name=test
> [junit4] 2> 89367 T740 oass.IndexSchema.readSchema default search field in
> schema is text
> [junit4] 2> 89370 T740 oass.IndexSchema.readSchema unique key field: id
> [junit4] 2> 89372 T740 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 89375 T740 oass.FileExchangeRateProvider.reload Reloading
> exchange rates from file currency.xml
> [junit4] 2> 89392 T740 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
> [junit4] 2> 89393 T740 oasc.SolrCore.<init> [collection1] Opening new
> SolrCore at
> ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/,
> dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/
> [junit4] 2> 89393 T740 oasc.JmxMonitoredMap.<init> JMX monitoring is
> enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
> [junit4] 2> 89394 T740 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
> [junit4] 2> 89395 T740 oasc.SolrCore.getNewIndexDir New index directory
> detected: old=null
> new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index/
> [junit4] 2> 89395 T740 oasc.SolrCore.initIndex WARN [collection1] Solr index
> directory
> './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index' doesn't
> exist. Creating new index...
> [junit4] 2> 89395 T740 oasc.CachingDirectoryFactory.get return new directory
> for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
> [junit4] 2> 89396 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true,
> maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12,
> noCFSRatio=1.0]
> [junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
> [junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "nodistrib"
> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "dedupe"
> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "stored_sig"
> [junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "stored_sig"
> [junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-explicit"
> [junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4] 2> 89404 T740 oasup.UpdateRequestProcessorChain.init inserting
> DistributedUpdateProcessorFactory into updateRequestProcessorChain
> "distrib-dup-test-chain-implicit"
> [junit4] 2> 89404 T740 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
> [junit4] 2> 89405 T740 oasc.RequestHandlers.initHandlersFromConfig created
> standard: solr.StandardRequestHandler
> [junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created
> /get: solr.RealTimeGetHandler
> [junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig adding
> lazy requestHandler: solr.ReplicationHandler
> [junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created
> /replication: solr.ReplicationHandler
> [junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created
> /update: solr.UpdateRequestHandler
> [junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4] 2> 89408 T740 oasc.RequestHandlers.initHandlersFromConfig created
> /admin/file: solr.admin.ShowFileRequestHandler
> [junit4] 2> 89411 T740 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4] 2> 89414 T740 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4] 2> 89414 T740 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4] 2> 89415 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy
> wrapping class org.apache.lucene.index.LogByteSizeMergePolicy:
> [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41,
> maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=false, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
> [junit4] 2> 89415 T740 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
> [junit4] 2> 89416 T740 oasc.SolrDeletionPolicy.updateCommits newest commit
> generation = 1
> [junit4] 2> 89416 T740 oass.SolrIndexSearcher.<init> Opening
> Searcher@c599fa[collection1] main
> [junit4] 2> 89422 T741 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@c599fa[collection1]
> main{StandardDirectoryReader(segments_1:1:nrt)}
> [junit4] 2> 89426 T740 oasc.CoreContainer.registerCore registering core:
> collection1
> [junit4] 2> 89426 T677 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4] 2> 89427 T677 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
> [junit4] 2> 89426 T744 oasc.ZkController.register Register replica -
> core:collection1 address:http://127.0.0.1:40129/nju/sc
> collection:collection1 shard:shard1
> [junit4] 2> 89428 T677 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 89430 T744 oasc.ZkController.register We are
> http://127.0.0.1:40129/nju/sc/collection1/ and leader is
> http://127.0.0.1:39738/nju/sc/collection1/
> [junit4] 2> 89430 T744 oasc.ZkController.register No LogReplay needed for
> core=collection1 baseURL=http://127.0.0.1:40129/nju/sc
> [junit4] 2> 89431 T744 oasc.ZkController.checkRecovery Core needs to
> recover:collection1
> [junit4] 2> 89431 T744 oasu.DefaultSolrCoreState.doRecovery Running recovery
> - first canceling any ongoing recovery
> [junit4] 2> 89431 T744 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> ASYNC NEW_CORE C598 name=collection1
> org.apache.solr.core.SolrCore@1ec0a26
> url=http://127.0.0.1:40129/nju/sc/collection1 node=127.0.0.1:40129_nju%2Fsc
> C598_STATE=coll:collection1 core:collection1 props:{state=down,
> base_url=http://127.0.0.1:40129/nju/sc, core=collection1,
> node_name=127.0.0.1:40129_nju%2Fsc}
> [junit4] 2> 89432 T747 C598 P40129 oasc.RecoveryStrategy.run Starting
> recovery process. core=collection1 recoveringAfterStartup=true
> [junit4] 2> 89433 T747 C598 P40129 oasc.RecoveryStrategy.doRecovery ######
> startupVersions=[]
> [junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish publishing
> core=collection1 state=recovering collection=collection1
> [junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
> [junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 89436 T700 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 89437 T714 oasha.CoreAdminHandler.handleWaitForStateAction Going
> to wait for coreNodeName: core_node2, state: recovering, checkLive: true,
> onlyIfLeader: true
> [junit4] 2> 89437 T700 oasc.Overseer$ClusterStateUpdater.updateState Update
> state numShards=1 message={
> [junit4] 2> "operation":"state",
> [junit4] 2> "state":"recovering",
> [junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
> [junit4] 2> "core":"collection1",
> [junit4] 2> "roles":null,
> [junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "numShards":"1",
> [junit4] 2> "core_node_name":"core_node2"}
> [junit4] 2> 89438 T714 oascc.ZkStateReader.updateClusterState Updating cloud
> state from ZooKeeper...
> [junit4] 2> 89439 T714 oasha.CoreAdminHandler.handleWaitForStateAction Will
> wait a max of 183 seconds to see collection1 (shard1 of collection1) have
> state: recovering
> [junit4] 2> 89439 T699 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 89440 T721 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 89440 T739 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 89440 T708 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> 89440 T699 oascc.ZkStateReader$2.process A cluster state change:
> WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4] 2> ASYNC NEW_CORE C599 name=collection1
> org.apache.solr.core.SolrCore@69814a
> url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc
> C599_STATE=coll:control_collection core:collection1 props:{state=active,
> base_url=http://127.0.0.1:38011/nju/sc, core=collection1,
> node_name=127.0.0.1:38011_nju%2Fsc, leader=true}
> [junit4] 2> 89447 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-0 (1463407845892423680)]}
> 0 14
> [junit4] 2> 89448 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-0 (1463407845892423681)]}
> 0 15
> [junit4] 2> C598_STATE=coll:collection1 core:collection1
> props:{state=recovering, base_url=http://127.0.0.1:40129/nju/sc,
> core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
> [junit4] 2> 89458 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-0 (1463407845911298048)]} 0 2
> [junit4] 2> 89458 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-0 (1463407845911298049)]} 0 2
> [junit4] 2> ASYNC NEW_CORE C600 name=collection1
> org.apache.solr.core.SolrCore@a6a2f3
> url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc
> C600_STATE=coll:collection1 core:collection1 props:{state=active,
> base_url=http://127.0.0.1:39738/nju/sc, core=collection1,
> node_name=127.0.0.1:39738_nju%2Fsc, leader=true}
> [junit4] 2> 89459 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-0 (1463407845911298048)]} 0 8
> [junit4] 2> 89460 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-0 (1463407845911298049)]} 0 9
> [junit4] 2> 89527 T693 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-0
> (-1463407845989941249)]} 0 1
> [junit4] 2> 89527 T695 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-0
> (-1463407845989941248)]} 0 1
> [junit4] 2> 89532 T730 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {delete=[2-0 (-1463407845994135553)]} 0 0
> [junit4] 2> 89532 T735 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {delete=[1-0 (-1463407845994135552)]} 0 0
> [junit4] 2> 89533 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {delete=[2-0 (-1463407845994135553)]} 0 3
> [junit4] 2> 89533 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {delete=[1-0 (-1463407845994135552)]} 0 3
> [junit4] 2> 89536 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-1 (1463407845999378432)]}
> 0 1
> [junit4] 2> 89536 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-1 (1463407845999378433)]}
> 0 1
> [junit4] 2> 89540 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-1 (1463407846002524160)]} 0 1
> [junit4] 2> 89540 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-1 (1463407846002524160)]} 0 2
> [junit4] 2> 89541 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-1 (1463407846002524161)]} 0 1
> [junit4] 2> 89541 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-1 (1463407846002524161)]} 0 3
> [junit4] 2> 89589 T693 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-2 (1463407846054952960)]}
> 0 1
> [junit4] 2> 89589 T695 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-2 (1463407846054952961)]}
> 0 1
> [junit4] 2> 89593 T730 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-2 (1463407846058098688)]} 0 0
> [junit4] 2> 89594 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-2 (1463407846058098688)]} 0 3
> [junit4] 2> 89594 T735 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-2 (1463407846058098689)]} 0 1
> [junit4] 2> 89595 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-2 (1463407846058098689)]} 0 4
> [junit4] 2> 89606 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-3 (1463407846072778752)]}
> 0 1
> [junit4] 2> 89606 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-3 (1463407846073827328)]}
> 0 0
> [junit4] 2> 89610 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-3 (1463407846075924480)]} 0 0
> [junit4] 2> 89611 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-3 (1463407846075924480)]} 0 3
> [junit4] 2> 89611 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-3 (1463407846075924481)]} 0 1
> [junit4] 2> 89611 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-3 (1463407846075924481)]} 0 3
> [junit4] 2> 89666 T693 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-4 (1463407846135693312)]}
> 0 1
> [junit4] 2> 89666 T695 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-4 (1463407846135693313)]}
> 0 1
> [junit4] 2> 89670 T730 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-4 (1463407846138839040)]} 0 1
> [junit4] 2> 89670 T735 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-4 (1463407846138839041)]} 0 0
> [junit4] 2> 89670 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-4 (1463407846138839040)]} 0 2
> [junit4] 2> 89671 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-4 (1463407846138839041)]} 0 3
> [junit4] 2> 89707 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-5 (1463407846179733505)]}
> 0 0
> [junit4] 2> 89708 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-5 (1463407846179733504)]}
> 0 1
> [junit4] 2> 89712 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-5 (1463407846181830656)]} 0 1
> [junit4] 2> 89712 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-5 (1463407846182879232)]} 0 1
> [junit4] 2> 89712 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-5 (1463407846181830656)]} 0 3
> [junit4] 2> 89713 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-5 (1463407846182879232)]} 0 3
> [junit4] 2> 89714 T693 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-2
> (-1463407846187073536)]} 0 0
> [junit4] 2> 89715 T695 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-2
> (-1463407846188122112)]} 0 0
> [junit4] 2> 89718 T730 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {delete=[2-2 (-1463407846190219264)]} 0 0
> [junit4] 2> 89718 T735 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {delete=[1-2 (-1463407846190219265)]} 0 0
> [junit4] 2> 89719 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {delete=[2-2 (-1463407846190219264)]} 0 2
> [junit4] 2> 89719 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {delete=[1-2 (-1463407846190219265)]} 0 2
> [junit4] 2> 89721 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-6 (1463407846194413568)]}
> 0 0
> [junit4] 2> 89721 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-6 (1463407846194413569)]}
> 0 0
> [junit4] 2> 89725 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-6 (1463407846196510720)]} 0 0
> [junit4] 2> 89725 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-6 (1463407846196510721)]} 0 0
> [junit4] 2> 89726 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-6 (1463407846196510720)]} 0 3
> [junit4] 2> 89726 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-6 (1463407846196510721)]} 0 3
> [junit4] 2> 89742 T693 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-7 (1463407846216433664)]}
> 0 0
> [junit4] 2> 89743 T695 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-7 (1463407846216433665)]}
> 0 1
> [junit4] 2> 89746 T730 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-7 (1463407846218530816)]} 0 0
> [junit4] 2> 89747 T735 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-7 (1463407846218530817)]} 0 1
> [junit4] 2> 89747 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-7 (1463407846218530816)]} 0 3
> [junit4] 2> 89747 T717 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-7 (1463407846218530817)]} 0 3
> [junit4] 2> 89794 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-8 (1463407846269911040)]}
> 0 1
> [junit4] 2> 89794 T694 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-8 (1463407846269911041)]}
> 0 1
> [junit4] 2> 89798 T733 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[2-8 (1463407846273056768)]} 0 1
> [junit4] 2> 89798 T734 C598 P40129 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER}
> {add=[1-8 (1463407846273056769)]} 0 0
> [junit4] 2> 89799 T715 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[2-8 (1463407846273056768)]} 0 3
> [junit4] 2> 89799 T716 C600 P39738 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin}
> {add=[1-8 (1463407846273056769)]} 0 3
> [junit4] 2> 89858 T690 C599 P38011 oasup.LogUpdateProcessor.finish
> [collection1] webapp=/nju/sc path=/update
> params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-9 (1463407846337019904)]}
> 0 1
> [junit
>
> [...truncated too long message...]
>
> asc.Overseer.createOverseerNode ERROR Could not create Overseer node
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
> [junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
> [junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> [junit4] 2>
> [junit4] 2> 191141 T765 oasc.SolrCore.close [collection1] CLOSING SolrCore
> org.apache.solr.core.SolrCore@966553
> [junit4] 2> 191156 T765 oasu.DirectUpdateHandler2.close closing
> DirectUpdateHandler2{commits=0,autocommits=0,soft
> autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=86,adds=86,deletesById=48,deletesByQuery=0,errors=0,cumulative_adds=86,cumulative_deletesById=48,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
> [junit4] 2> 191156 T765 oasu.SolrCoreState.decrefSolrCoreState Closing
> SolrCoreState
> [junit4] 2> 191156 T765 oasu.DefaultSolrCoreState.closeIndexWriter
> SolrCoreState ref count has reached 0 - closing IndexWriter
> [junit4] 2> 191157 T765 oasu.DefaultSolrCoreState.closeIndexWriter closing
> IndexWriter with IndexWriterCloser
> [junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=2
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_1,generation=1}
> [junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6
> lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_2,generation=2}
> [junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 2
> [junit4] 2> 191162 T765 oasc.SolrCore.closeSearcher [collection1] Closing
> main searcher on request.
> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing
> MockDirectoryFactory - 2 directories currently being tracked
> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking
> to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
> [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2;done=false>>]
> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing
> directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking
> to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
> [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index;done=false>>]
> [junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing
> directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
> [junit4] 2> 191164 T765 oasc.LeaderElector$1.process WARN
> java.lang.RuntimeException:
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer
> [junit4] 2> at
> org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1227)
> [junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
> [junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
> [junit4] 2> at
> org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
> [junit4] 2> at
> org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> [junit4] 2> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> [junit4] 2> Caused by:
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /overseer
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> [junit4] 2> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> [junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
> [junit4] 2> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
> [junit4] 2> at
> org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
> [junit4] 2> at
> org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
> [junit4] 2> ... 14 more
> [junit4] 2>
> [junit4] 2> 191164 T765 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 191164 T765 oascc.ZkStateReader$2.process A cluster state
> change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4] 2> 191165 T765 oascc.ZkStateReader$2.process WARN ZooKeeper watch
> triggered, but Solr cannot talk to ZK
> [junit4] 2> 191165 T765 oasc.DistributedQueue$LatchChildWatcher.process
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> [junit4] 2> 191165 T765 oascc.ZkStateReader$3.process WARN ZooKeeper watch
> triggered, but Solr cannot talk to ZK
> [junit4] 2> NOTE: test params are: codec=Lucene42,
> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_GT,
> timezone=Asia/Brunei
> [junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51
> (32-bit)/cpus=8,threads=1,free=80585760,total=235929600
> [junit4] 2> NOTE: All tests run in this JVM: [DistribCursorPagingTest,
> ParsingFieldUpdateProcessorsTest, OverseerTest, RecoveryZkTest]
> [junit4] Completed on J0 in 111.30s, 1 test, 1 failure <<< FAILURES!
>
> [...truncated 1081 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following
> error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908:
> There were test failures: 378 suites, 1598 tests, 1 failure, 37 ignored (16
> assumptions)
>
> Total time: 58 minutes 22 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build
# 9780 - Still Failing!
Posted by Mark Miller <ma...@gmail.com>.
Issue is that dir.openInput(fileName, IOContext.DEFAULT).close() is not a perfect replacement for Directory#fileExists when using the MockDirectoryWrapper - if the file is still being written, it throws an IOException.Â
I don’t have a solution I’m in love with yet.
Could special case it for MockDirectoryWrapper - not super appealing. Might be better if we make some new exceptions though.
Could assume the file exists on IOException or IOException caused by RuntimeException - not very appealing.
Could disable that check for Solr tests that replicate - not very appealing.
Could stop using MockDirectoryWrapper on those tests - not very appealing.
--Â
Mark Miller
about.me/markrmiller
On March 23, 2014 at 7:48:47 PM, Mark Miller (markrmiller@gmail.com) wrote:
Okay, looks LUCENE-5543
related. I’ll dig into it shortly.
--Â
Mark Miller
about.me/markrmiller
On March 23, 2014 at 7:44:17 PM, Mark Miller (markrmiller@gmail.com) wrote:
Cause of this seems to be:
24422 T107 C8 P54679 oasc.SolrException.log ERROR SnapPull failed :org.apache.solr.common.SolrException: Index fetch failed :Â
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:506)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:324)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:446)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:234)
Caused by: java.io.IOException: MockDirectoryWrapper: file "_0.fdx" is still open for writing
at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:568)
at org.apache.solr.handler.SnapPuller.slowFileExists(SnapPuller.java:789)
at org.apache.solr.handler.SnapPuller.isIndexStale(SnapPuller.java:805)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:405)
... 4 more
Caused by: java.lang.RuntimeException: unclosed IndexOutput: _0.fdx
at org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:544)
at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:516)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:109)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120)
at org.apache.lucene.index.StoredFieldsProcessor.initFieldsWriter(StoredFieldsProcessor.java:88)
at org.apache.lucene.index.StoredFieldsProcessor.finishDocument(StoredFieldsProcessor.java:120)
at org.apache.lucene.index.TwoStoredFieldsConsumers.finishDocument(TwoStoredFieldsConsumers.java:65)
at org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:273)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:271)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:465)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1536)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:236)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:704)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:858)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:557)
at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1957)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:785)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:429)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:134)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
--Â
Mark Miller
about.me/markrmiller
On March 23, 2014 at 7:04:59 PM, Policeman Jenkins Server (jenkins@thetaphi.de) wrote:
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9780/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
There are still nodes recoverying - waited for 90 seconds
Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 90 seconds
at __randomizedtesting.SeedInfo.seed([92A83ABF6D073DEE:134EB4A71A585DD2]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:177)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:732)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1495)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:103)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Build Log:
[...truncated 10169 lines...]
[junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4] 2> 80886 T677 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 80887 T677 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /nju/sc
[junit4] 2> 80892 T677 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1395614468515
[junit4] 2> 80893 T677 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 80894 T678 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 80994 T677 oasc.ZkTestServer.run start zk server on port:46640
[junit4] 2> 80995 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81204 T684 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@afcd05 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81204 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81205 T677 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 81209 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81210 T686 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1598a49 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81210 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81211 T677 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 81213 T677 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 81215 T677 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 81216 T677 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 81218 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 81218 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 81220 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 81221 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 81224 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 81224 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 81226 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 81226 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 81228 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 81228 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 81230 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 81230 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 81231 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 81232 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 81233 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 81234 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 81235 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 81236 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 81237 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 81238 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 81239 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 81240 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 81475 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 81479 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38011
[junit4] 2> 81480 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864
[junit4] 2> 81482 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/'
[junit4] 2> 81536 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/solr.xml
[junit4] 2> 81599 T677 oasc.CoreContainer.<init> New CoreContainer 27496700
[junit4] 2> 81599 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/]
[junit4] 2> 81600 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 81603 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 81603 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 81604 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 81604 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 81604 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 81605 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 81605 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81607 T697 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1839970 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81607 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81609 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81609 T679 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144f11ac9a60002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 81611 T699 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6336bf name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81611 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81612 T677 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 81614 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 81616 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 81618 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 81620 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 81621 T677 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 81623 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81625 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81627 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 81628 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 81631 T677 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81631 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 81632 T677 oasc.Overseer.start Overseer (id=91462989808795651-127.0.0.1:38011_nju%2Fsc-n_0000000000) starting
[junit4] 2> 81637 T677 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 81645 T701 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 81645 T677 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 81646 T701 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 81647 T677 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 81648 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 81650 T700 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 81655 T702 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 81656 T702 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 81657 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 81657 T702 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 81658 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 81661 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 81662 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 82658 T702 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 82658 T702 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1
[junit4] 2> 82658 T702 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 82659 T702 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 82659 T702 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 82660 T702 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 82660 T702 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/'
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/.svn/' to classloader
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/classes/' to classloader
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/README' to classloader
[junit4] 2> 82714 T702 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 82762 T702 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 82763 T702 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 82788 T702 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 83255 T702 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 83258 T702 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 83260 T702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 83264 T702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 83285 T702 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 83286 T702 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/
[junit4] 2> 83289 T702 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 83292 T702 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data
[junit4] 2> 83292 T702 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index/
[junit4] 2> 83292 T702 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 83293 T702 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index
[junit4] 2> 83293 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
[junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 83299 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 83310 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 83313 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 83338 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 83339 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 83339 T702 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 83341 T702 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 83344 T702 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 83349 T702 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 83351 T702 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 83355 T702 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 83355 T702 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 83357 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
[junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 83358 T702 oass.SolrIndexSearcher.<init> Opening Searcher@1c23829[collection1] main
[junit4] 2> 83365 T703 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c23829[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 83368 T702 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 83369 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 83369 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 83369 T706 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38011/nju/sc collection:control_collection shard:shard1
[junit4] 2> 83370 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83371 T706 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 83372 T708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9657d4 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83372 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83373 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 83375 T677 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 83376 T706 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 83377 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C596 name=collection1 org.apache.solr.core.SolrCore@69814a url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc C596_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:38011/nju/sc, core=collection1, node_name=127.0.0.1:38011_nju%2Fsc}
[junit4] 2> 83378 T706 C596 P38011 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38011/nju/sc/collection1/
[junit4] 2> 83379 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncToMe http://127.0.0.1:38011/nju/sc/collection1/ has no replicas
[junit4] 2> 83379 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38011/nju/sc/collection1/ shard1
[junit4] 2> 83380 T706 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 83381 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83382 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83382 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83385 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83386 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83490 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83498 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83539 T706 oasc.ZkController.register We are http://127.0.0.1:38011/nju/sc/collection1/ and leader is http://127.0.0.1:38011/nju/sc/collection1/
[junit4] 2> 83539 T706 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38011/nju/sc
[junit4] 2> 83539 T706 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 83540 T706 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 83540 T706 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83542 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83542 T706 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83543 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83544 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 83546 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83631 T677 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 83631 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 83651 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39738
[junit4] 2> 83652 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998
[junit4] 2> 83655 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/'
[junit4] 2> 83655 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83669 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/solr.xml
[junit4] 2> 83720 T677 oasc.CoreContainer.<init> New CoreContainer 29042762
[junit4] 2> 83721 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/]
[junit4] 2> 83722 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 83725 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 83725 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 83726 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 83726 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 83726 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 83727 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 83727 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83729 T719 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5231ef name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83729 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83731 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83732 T721 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1075f95 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83733 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83737 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 84742 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39738_nju%2Fsc
[junit4] 2> 84743 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39738_nju%2Fsc
[junit4] 2> 84745 T708 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84745 T721 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84745 T699 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84751 T722 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 84752 T722 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84755 T722 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84756 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84757 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 84758 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 84759 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84862 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 84862 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 84862 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 85756 T722 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 85756 T722 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1
[junit4] 2> 85756 T722 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 85757 T722 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 85758 T722 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 85759 T722 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 85759 T722 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/'
[junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/.svn/' to classloader
[junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/classes/' to classloader
[junit4] 2> 85761 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/README' to classloader
[junit4] 2> 85807 T722 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 85855 T722 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 85856 T722 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 85866 T722 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 86312 T722 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 86315 T722 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 86317 T722 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 86321 T722 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 86342 T722 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 86343 T722 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/
[junit4] 2> 86343 T722 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 86346 T722 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1
[junit4] 2> 86346 T722 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index/
[junit4] 2> 86346 T722 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 86347 T722 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index
[junit4] 2> 86347 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
[junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 86353 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 86356 T722 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 86357 T722 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 86358 T722 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 86361 T722 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 86364 T722 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 86367 T722 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 86367 T722 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 86368 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
[junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 86369 T722 oass.SolrIndexSearcher.<init> Opening Searcher@383c54[collection1] main
[junit4] 2> 86376 T723 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@383c54[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 86381 T722 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 86382 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 86382 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 86382 T726 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39738/nju/sc collection:collection1 shard:shard1
[junit4] 2> 86383 T726 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 86387 T726 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C597 name=collection1 org.apache.solr.core.SolrCore@a6a2f3 url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc C597_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:39738/nju/sc, core=collection1, node_name=127.0.0.1:39738_nju%2Fsc}
[junit4] 2> 86389 T726 C597 P39738 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 86390 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86390 T726 C597 P39738 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 86391 T726 C597 P39738 oasc.SyncStrategy.syncToMe http://127.0.0.1:39738/nju/sc/collection1/ has no replicas
[junit4] 2> 86390 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86391 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39738/nju/sc/collection1/ shard1
[junit4] 2> 86391 T726 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 86392 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86394 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86394 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86394 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86397 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86399 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86501 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86501 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86501 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86547 T726 oasc.ZkController.register We are http://127.0.0.1:39738/nju/sc/collection1/ and leader is http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 86547 T726 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39738/nju/sc
[junit4] 2> 86547 T726 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 86548 T726 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 86548 T726 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86550 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86550 T726 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86551 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86551 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 86553 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86655 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86655 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86655 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86667 T677 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 86668 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 86670 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40129
[junit4] 2> 86671 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 86671 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 86672 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005
[junit4] 2> 86672 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/'
[junit4] 2> 86704 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/solr.xml
[junit4] 2> 86751 T677 oasc.CoreContainer.<init> New CoreContainer 10907151
[junit4] 2> 86751 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/]
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 86756 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 86756 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 86756 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 86757 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 86757 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 86758 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 86759 T737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@83c633 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 86760 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 86762 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 86763 T739 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18288ea name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 86763 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 86768 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 87770 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40129_nju%2Fsc
[junit4] 2> 87771 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40129_nju%2Fsc
[junit4] 2> 87773 T699 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87773 T708 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87773 T721 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87774 T739 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87781 T740 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 87781 T740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 87783 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87783 T740 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 87785 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 87789 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87890 T739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 88785 T740 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 88785 T740 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1
[junit4] 2> 88785 T740 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 88786 T740 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 88786 T740 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 88787 T740 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 88787 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/'
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/.svn/' to classloader
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/classes/' to classloader
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/README' to classloader
[junit4] 2> 88833 T740 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 88882 T740 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 88883 T740 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 88895 T740 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 89367 T740 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 89370 T740 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 89372 T740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 89375 T740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 89392 T740 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 89393 T740 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/
[junit4] 2> 89393 T740 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 89394 T740 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
[junit4] 2> 89395 T740 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index/
[junit4] 2> 89395 T740 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 89395 T740 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
[junit4] 2> 89396 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
[junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 89404 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 89404 T740 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 89405 T740 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 89408 T740 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 89411 T740 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 89414 T740 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 89414 T740 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 89415 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 89415 T740 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
[junit4] 2> 89416 T740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 89416 T740 oass.SolrIndexSearcher.<init> Opening Searcher@c599fa[collection1] main
[junit4] 2> 89422 T741 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c599fa[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 89426 T740 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 89426 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 89427 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 89426 T744 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40129/nju/sc collection:collection1 shard:shard1
[junit4] 2> 89428 T677 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89430 T744 oasc.ZkController.register We are http://127.0.0.1:40129/nju/sc/collection1/ and leader is http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 89430 T744 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40129/nju/sc
[junit4] 2> 89431 T744 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 89431 T744 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> 89431 T744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C598 name=collection1 org.apache.solr.core.SolrCore@1ec0a26 url=http://127.0.0.1:40129/nju/sc/collection1 node=127.0.0.1:40129_nju%2Fsc C598_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:40129/nju/sc, core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
[junit4] 2> 89432 T747 C598 P40129 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 89433 T747 C598 P40129 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
[junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89436 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89437 T714 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 89437 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 89438 T714 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89439 T714 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 89439 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89440 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> ASYNC NEW_CORE C599 name=collection1 org.apache.solr.core.SolrCore@69814a url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc C599_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:38011/nju/sc, core=collection1, node_name=127.0.0.1:38011_nju%2Fsc, leader=true}
[junit4] 2> 89447 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-0 (1463407845892423680)]} 0 14
[junit4] 2> 89448 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-0 (1463407845892423681)]} 0 15
[junit4] 2> C598_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=http://127.0.0.1:40129/nju/sc, core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
[junit4] 2> 89458 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-0 (1463407845911298048)]} 0 2
[junit4] 2> 89458 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-0 (1463407845911298049)]} 0 2
[junit4] 2> ASYNC NEW_CORE C600 name=collection1 org.apache.solr.core.SolrCore@a6a2f3 url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc C600_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:39738/nju/sc, core=collection1, node_name=127.0.0.1:39738_nju%2Fsc, leader=true}
[junit4] 2> 89459 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-0 (1463407845911298048)]} 0 8
[junit4] 2> 89460 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-0 (1463407845911298049)]} 0 9
[junit4] 2> 89527 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-0 (-1463407845989941249)]} 0 1
[junit4] 2> 89527 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-0 (-1463407845989941248)]} 0 1
[junit4] 2> 89532 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[2-0 (-1463407845994135553)]} 0 0
[junit4] 2> 89532 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[1-0 (-1463407845994135552)]} 0 0
[junit4] 2> 89533 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[2-0 (-1463407845994135553)]} 0 3
[junit4] 2> 89533 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[1-0 (-1463407845994135552)]} 0 3
[junit4] 2> 89536 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-1 (1463407845999378432)]} 0 1
[junit4] 2> 89536 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-1 (1463407845999378433)]} 0 1
[junit4] 2> 89540 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-1 (1463407846002524160)]} 0 1
[junit4] 2> 89540 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-1 (1463407846002524160)]} 0 2
[junit4] 2> 89541 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-1 (1463407846002524161)]} 0 1
[junit4] 2> 89541 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-1 (1463407846002524161)]} 0 3
[junit4] 2> 89589 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-2 (1463407846054952960)]} 0 1
[junit4] 2> 89589 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-2 (1463407846054952961)]} 0 1
[junit4] 2> 89593 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-2 (1463407846058098688)]} 0 0
[junit4] 2> 89594 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-2 (1463407846058098688)]} 0 3
[junit4] 2> 89594 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-2 (1463407846058098689)]} 0 1
[junit4] 2> 89595 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-2 (1463407846058098689)]} 0 4
[junit4] 2> 89606 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-3 (1463407846072778752)]} 0 1
[junit4] 2> 89606 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-3 (1463407846073827328)]} 0 0
[junit4] 2> 89610 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-3 (1463407846075924480)]} 0 0
[junit4] 2> 89611 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-3 (1463407846075924480)]} 0 3
[junit4] 2> 89611 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-3 (1463407846075924481)]} 0 1
[junit4] 2> 89611 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-3 (1463407846075924481)]} 0 3
[junit4] 2> 89666 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-4 (1463407846135693312)]} 0 1
[junit4] 2> 89666 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-4 (1463407846135693313)]} 0 1
[junit4] 2> 89670 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-4 (1463407846138839040)]} 0 1
[junit4] 2> 89670 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-4 (1463407846138839041)]} 0 0
[junit4] 2> 89670 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-4 (1463407846138839040)]} 0 2
[junit4] 2> 89671 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-4 (1463407846138839041)]} 0 3
[junit4] 2> 89707 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-5 (1463407846179733505)]} 0 0
[junit4] 2> 89708 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-5 (1463407846179733504)]} 0 1
[junit4] 2> 89712 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-5 (1463407846181830656)]} 0 1
[junit4] 2> 89712 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-5 (1463407846182879232)]} 0 1
[junit4] 2> 89712 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-5 (1463407846181830656)]} 0 3
[junit4] 2> 89713 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-5 (1463407846182879232)]} 0 3
[junit4] 2> 89714 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-2 (-1463407846187073536)]} 0 0
[junit4] 2> 89715 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-2 (-1463407846188122112)]} 0 0
[junit4] 2> 89718 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[2-2 (-1463407846190219264)]} 0 0
[junit4] 2> 89718 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[1-2 (-1463407846190219265)]} 0 0
[junit4] 2> 89719 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[2-2 (-1463407846190219264)]} 0 2
[junit4] 2> 89719 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[1-2 (-1463407846190219265)]} 0 2
[junit4] 2> 89721 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-6 (1463407846194413568)]} 0 0
[junit4] 2> 89721 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-6 (1463407846194413569)]} 0 0
[junit4] 2> 89725 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-6 (1463407846196510720)]} 0 0
[junit4] 2> 89725 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-6 (1463407846196510721)]} 0 0
[junit4] 2> 89726 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-6 (1463407846196510720)]} 0 3
[junit4] 2> 89726 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-6 (1463407846196510721)]} 0 3
[junit4] 2> 89742 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-7 (1463407846216433664)]} 0 0
[junit4] 2> 89743 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-7 (1463407846216433665)]} 0 1
[junit4] 2> 89746 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-7 (1463407846218530816)]} 0 0
[junit4] 2> 89747 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-7 (1463407846218530817)]} 0 1
[junit4] 2> 89747 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-7 (1463407846218530816)]} 0 3
[junit4] 2> 89747 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-7 (1463407846218530817)]} 0 3
[junit4] 2> 89794 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-8 (1463407846269911040)]} 0 1
[junit4] 2> 89794 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-8 (1463407846269911041)]} 0 1
[junit4] 2> 89798 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-8 (1463407846273056768)]} 0 1
[junit4] 2> 89798 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-8 (1463407846273056769)]} 0 0
[junit4] 2> 89799 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-8 (1463407846273056768)]} 0 3
[junit4] 2> 89799 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-8 (1463407846273056769)]} 0 3
[junit4] 2> 89858 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-9 (1463407846337019904)]} 0 1
[junit
[...truncated too long message...]
asc.Overseer.createOverseerNode ERROR Could not create Overseer node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 191141 T765 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@966553
[junit4] 2> 191156 T765 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=86,adds=86,deletesById=48,deletesByQuery=0,errors=0,cumulative_adds=86,cumulative_deletesById=48,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
[junit4] 2> 191156 T765 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 191156 T765 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 191157 T765 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_1,generation=1}
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_2,generation=2}
[junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4] 2> 191162 T765 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2;done=false>>]
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index;done=false>>]
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
[junit4] 2> 191164 T765 oasc.LeaderElector$1.process WARN java.lang.RuntimeException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1227)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
[junit4] 2> ... 14 more
[junit4] 2>
[junit4] 2> 191164 T765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 191164 T765 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 191165 T765 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 191165 T765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 191165 T765 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_GT, timezone=Asia/Brunei
[junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=80585760,total=235929600
[junit4] 2> NOTE: All tests run in this JVM: [DistribCursorPagingTest, ParsingFieldUpdateProcessorsTest, OverseerTest, RecoveryZkTest]
[junit4] Completed on J0 in 111.30s, 1 test, 1 failure <<< FAILURES!
[...truncated 1081 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908: There were test failures: 378 suites, 1598 tests, 1 failure, 37 ignored (16 assumptions)
Total time: 58 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build
# 9780 - Still Failing!
Posted by Mark Miller <ma...@gmail.com>.
Okay, looks LUCENE-5543 related. I’ll dig into it shortly.
--Â
Mark Miller
about.me/markrmiller
On March 23, 2014 at 7:44:17 PM, Mark Miller (markrmiller@gmail.com) wrote:
Cause of this seems to be:
24422 T107 C8 P54679 oasc.SolrException.log ERROR SnapPull failed :org.apache.solr.common.SolrException: Index fetch failed :Â
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:506)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:324)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:446)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:234)
Caused by: java.io.IOException: MockDirectoryWrapper: file "_0.fdx" is still open for writing
at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:568)
at org.apache.solr.handler.SnapPuller.slowFileExists(SnapPuller.java:789)
at org.apache.solr.handler.SnapPuller.isIndexStale(SnapPuller.java:805)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:405)
... 4 more
Caused by: java.lang.RuntimeException: unclosed IndexOutput: _0.fdx
at org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:544)
at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:516)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:109)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120)
at org.apache.lucene.index.StoredFieldsProcessor.initFieldsWriter(StoredFieldsProcessor.java:88)
at org.apache.lucene.index.StoredFieldsProcessor.finishDocument(StoredFieldsProcessor.java:120)
at org.apache.lucene.index.TwoStoredFieldsConsumers.finishDocument(TwoStoredFieldsConsumers.java:65)
at org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:273)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:271)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:465)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1536)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:236)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:704)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:858)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:557)
at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1957)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:785)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:429)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:134)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
--Â
Mark Miller
about.me/markrmiller
On March 23, 2014 at 7:04:59 PM, Policeman Jenkins Server (jenkins@thetaphi.de) wrote:
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9780/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
There are still nodes recoverying - waited for 90 seconds
Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 90 seconds
at __randomizedtesting.SeedInfo.seed([92A83ABF6D073DEE:134EB4A71A585DD2]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:177)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:732)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1495)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:103)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Build Log:
[...truncated 10169 lines...]
[junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4] 2> 80886 T677 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 80887 T677 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /nju/sc
[junit4] 2> 80892 T677 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1395614468515
[junit4] 2> 80893 T677 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 80894 T678 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 80994 T677 oasc.ZkTestServer.run start zk server on port:46640
[junit4] 2> 80995 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81204 T684 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@afcd05 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81204 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81205 T677 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 81209 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81210 T686 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1598a49 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81210 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81211 T677 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 81213 T677 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 81215 T677 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 81216 T677 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 81218 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 81218 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 81220 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 81221 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 81224 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 81224 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 81226 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 81226 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 81228 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 81228 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 81230 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 81230 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 81231 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 81232 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 81233 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 81234 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 81235 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 81236 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 81237 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 81238 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 81239 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 81240 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 81475 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 81479 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38011
[junit4] 2> 81480 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864
[junit4] 2> 81482 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/'
[junit4] 2> 81536 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/solr.xml
[junit4] 2> 81599 T677 oasc.CoreContainer.<init> New CoreContainer 27496700
[junit4] 2> 81599 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/]
[junit4] 2> 81600 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 81603 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 81603 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 81604 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 81604 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 81604 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 81605 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 81605 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81607 T697 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1839970 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81607 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81609 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81609 T679 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144f11ac9a60002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 81611 T699 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6336bf name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81611 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81612 T677 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 81614 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 81616 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 81618 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 81620 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 81621 T677 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 81623 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81625 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81627 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 81628 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 81631 T677 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81631 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 81632 T677 oasc.Overseer.start Overseer (id=91462989808795651-127.0.0.1:38011_nju%2Fsc-n_0000000000) starting
[junit4] 2> 81637 T677 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 81645 T701 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 81645 T677 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 81646 T701 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 81647 T677 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 81648 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 81650 T700 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 81655 T702 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 81656 T702 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 81657 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 81657 T702 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 81658 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 81661 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 81662 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 82658 T702 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 82658 T702 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1
[junit4] 2> 82658 T702 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 82659 T702 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 82659 T702 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 82660 T702 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 82660 T702 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/'
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/.svn/' to classloader
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/classes/' to classloader
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/README' to classloader
[junit4] 2> 82714 T702 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 82762 T702 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 82763 T702 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 82788 T702 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 83255 T702 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 83258 T702 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 83260 T702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 83264 T702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 83285 T702 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 83286 T702 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/
[junit4] 2> 83289 T702 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 83292 T702 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data
[junit4] 2> 83292 T702 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index/
[junit4] 2> 83292 T702 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 83293 T702 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index
[junit4] 2> 83293 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
[junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 83299 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 83310 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 83313 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 83338 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 83339 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 83339 T702 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 83341 T702 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 83344 T702 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 83349 T702 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 83351 T702 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 83355 T702 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 83355 T702 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 83357 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
[junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 83358 T702 oass.SolrIndexSearcher.<init> Opening Searcher@1c23829[collection1] main
[junit4] 2> 83365 T703 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c23829[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 83368 T702 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 83369 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 83369 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 83369 T706 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38011/nju/sc collection:control_collection shard:shard1
[junit4] 2> 83370 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83371 T706 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 83372 T708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9657d4 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83372 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83373 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 83375 T677 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 83376 T706 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 83377 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C596 name=collection1 org.apache.solr.core.SolrCore@69814a url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc C596_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:38011/nju/sc, core=collection1, node_name=127.0.0.1:38011_nju%2Fsc}
[junit4] 2> 83378 T706 C596 P38011 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38011/nju/sc/collection1/
[junit4] 2> 83379 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncToMe http://127.0.0.1:38011/nju/sc/collection1/ has no replicas
[junit4] 2> 83379 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38011/nju/sc/collection1/ shard1
[junit4] 2> 83380 T706 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 83381 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83382 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83382 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83385 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83386 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83490 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83498 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83539 T706 oasc.ZkController.register We are http://127.0.0.1:38011/nju/sc/collection1/ and leader is http://127.0.0.1:38011/nju/sc/collection1/
[junit4] 2> 83539 T706 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38011/nju/sc
[junit4] 2> 83539 T706 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 83540 T706 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 83540 T706 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83542 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83542 T706 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83543 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83544 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 83546 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83631 T677 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 83631 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 83651 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39738
[junit4] 2> 83652 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998
[junit4] 2> 83655 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/'
[junit4] 2> 83655 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83669 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/solr.xml
[junit4] 2> 83720 T677 oasc.CoreContainer.<init> New CoreContainer 29042762
[junit4] 2> 83721 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/]
[junit4] 2> 83722 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 83725 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 83725 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 83726 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 83726 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 83726 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 83727 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 83727 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83729 T719 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5231ef name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83729 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83731 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83732 T721 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1075f95 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83733 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83737 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 84742 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39738_nju%2Fsc
[junit4] 2> 84743 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39738_nju%2Fsc
[junit4] 2> 84745 T708 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84745 T721 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84745 T699 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84751 T722 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 84752 T722 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84755 T722 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84756 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84757 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 84758 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 84759 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84862 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 84862 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 84862 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 85756 T722 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 85756 T722 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1
[junit4] 2> 85756 T722 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 85757 T722 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 85758 T722 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 85759 T722 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 85759 T722 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/'
[junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/.svn/' to classloader
[junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/classes/' to classloader
[junit4] 2> 85761 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/README' to classloader
[junit4] 2> 85807 T722 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 85855 T722 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 85856 T722 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 85866 T722 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 86312 T722 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 86315 T722 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 86317 T722 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 86321 T722 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 86342 T722 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 86343 T722 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/
[junit4] 2> 86343 T722 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 86346 T722 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1
[junit4] 2> 86346 T722 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index/
[junit4] 2> 86346 T722 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 86347 T722 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index
[junit4] 2> 86347 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
[junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 86353 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 86356 T722 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 86357 T722 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 86358 T722 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 86361 T722 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 86364 T722 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 86367 T722 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 86367 T722 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 86368 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
[junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 86369 T722 oass.SolrIndexSearcher.<init> Opening Searcher@383c54[collection1] main
[junit4] 2> 86376 T723 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@383c54[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 86381 T722 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 86382 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 86382 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 86382 T726 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39738/nju/sc collection:collection1 shard:shard1
[junit4] 2> 86383 T726 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 86387 T726 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C597 name=collection1 org.apache.solr.core.SolrCore@a6a2f3 url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc C597_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:39738/nju/sc, core=collection1, node_name=127.0.0.1:39738_nju%2Fsc}
[junit4] 2> 86389 T726 C597 P39738 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 86390 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86390 T726 C597 P39738 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 86391 T726 C597 P39738 oasc.SyncStrategy.syncToMe http://127.0.0.1:39738/nju/sc/collection1/ has no replicas
[junit4] 2> 86390 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86391 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39738/nju/sc/collection1/ shard1
[junit4] 2> 86391 T726 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 86392 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86394 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86394 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86394 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86397 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86399 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86501 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86501 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86501 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86547 T726 oasc.ZkController.register We are http://127.0.0.1:39738/nju/sc/collection1/ and leader is http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 86547 T726 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39738/nju/sc
[junit4] 2> 86547 T726 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 86548 T726 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 86548 T726 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86550 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86550 T726 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86551 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86551 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 86553 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86655 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86655 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86655 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86667 T677 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 86668 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 86670 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40129
[junit4] 2> 86671 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 86671 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 86672 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005
[junit4] 2> 86672 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/'
[junit4] 2> 86704 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/solr.xml
[junit4] 2> 86751 T677 oasc.CoreContainer.<init> New CoreContainer 10907151
[junit4] 2> 86751 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/]
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 86756 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 86756 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 86756 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 86757 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 86757 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 86758 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 86759 T737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@83c633 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 86760 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 86762 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 86763 T739 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18288ea name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 86763 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 86768 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 87770 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40129_nju%2Fsc
[junit4] 2> 87771 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40129_nju%2Fsc
[junit4] 2> 87773 T699 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87773 T708 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87773 T721 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87774 T739 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87781 T740 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 87781 T740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 87783 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87783 T740 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 87785 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 87789 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87890 T739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 88785 T740 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 88785 T740 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1
[junit4] 2> 88785 T740 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 88786 T740 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 88786 T740 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 88787 T740 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 88787 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/'
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/.svn/' to classloader
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/classes/' to classloader
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/README' to classloader
[junit4] 2> 88833 T740 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 88882 T740 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 88883 T740 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 88895 T740 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 89367 T740 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 89370 T740 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 89372 T740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 89375 T740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 89392 T740 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 89393 T740 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/
[junit4] 2> 89393 T740 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 89394 T740 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
[junit4] 2> 89395 T740 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index/
[junit4] 2> 89395 T740 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 89395 T740 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
[junit4] 2> 89396 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
[junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 89404 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 89404 T740 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 89405 T740 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 89408 T740 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 89411 T740 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 89414 T740 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 89414 T740 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 89415 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 89415 T740 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
[junit4] 2> 89416 T740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 89416 T740 oass.SolrIndexSearcher.<init> Opening Searcher@c599fa[collection1] main
[junit4] 2> 89422 T741 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c599fa[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 89426 T740 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 89426 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 89427 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 89426 T744 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40129/nju/sc collection:collection1 shard:shard1
[junit4] 2> 89428 T677 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89430 T744 oasc.ZkController.register We are http://127.0.0.1:40129/nju/sc/collection1/ and leader is http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 89430 T744 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40129/nju/sc
[junit4] 2> 89431 T744 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 89431 T744 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> 89431 T744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C598 name=collection1 org.apache.solr.core.SolrCore@1ec0a26 url=http://127.0.0.1:40129/nju/sc/collection1 node=127.0.0.1:40129_nju%2Fsc C598_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:40129/nju/sc, core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
[junit4] 2> 89432 T747 C598 P40129 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 89433 T747 C598 P40129 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
[junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89436 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89437 T714 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 89437 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 89438 T714 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89439 T714 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 89439 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89440 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> ASYNC NEW_CORE C599 name=collection1 org.apache.solr.core.SolrCore@69814a url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc C599_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:38011/nju/sc, core=collection1, node_name=127.0.0.1:38011_nju%2Fsc, leader=true}
[junit4] 2> 89447 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-0 (1463407845892423680)]} 0 14
[junit4] 2> 89448 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-0 (1463407845892423681)]} 0 15
[junit4] 2> C598_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=http://127.0.0.1:40129/nju/sc, core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
[junit4] 2> 89458 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-0 (1463407845911298048)]} 0 2
[junit4] 2> 89458 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-0 (1463407845911298049)]} 0 2
[junit4] 2> ASYNC NEW_CORE C600 name=collection1 org.apache.solr.core.SolrCore@a6a2f3 url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc C600_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:39738/nju/sc, core=collection1, node_name=127.0.0.1:39738_nju%2Fsc, leader=true}
[junit4] 2> 89459 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-0 (1463407845911298048)]} 0 8
[junit4] 2> 89460 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-0 (1463407845911298049)]} 0 9
[junit4] 2> 89527 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-0 (-1463407845989941249)]} 0 1
[junit4] 2> 89527 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-0 (-1463407845989941248)]} 0 1
[junit4] 2> 89532 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[2-0 (-1463407845994135553)]} 0 0
[junit4] 2> 89532 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[1-0 (-1463407845994135552)]} 0 0
[junit4] 2> 89533 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[2-0 (-1463407845994135553)]} 0 3
[junit4] 2> 89533 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[1-0 (-1463407845994135552)]} 0 3
[junit4] 2> 89536 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-1 (1463407845999378432)]} 0 1
[junit4] 2> 89536 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-1 (1463407845999378433)]} 0 1
[junit4] 2> 89540 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-1 (1463407846002524160)]} 0 1
[junit4] 2> 89540 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-1 (1463407846002524160)]} 0 2
[junit4] 2> 89541 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-1 (1463407846002524161)]} 0 1
[junit4] 2> 89541 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-1 (1463407846002524161)]} 0 3
[junit4] 2> 89589 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-2 (1463407846054952960)]} 0 1
[junit4] 2> 89589 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-2 (1463407846054952961)]} 0 1
[junit4] 2> 89593 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-2 (1463407846058098688)]} 0 0
[junit4] 2> 89594 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-2 (1463407846058098688)]} 0 3
[junit4] 2> 89594 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-2 (1463407846058098689)]} 0 1
[junit4] 2> 89595 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-2 (1463407846058098689)]} 0 4
[junit4] 2> 89606 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-3 (1463407846072778752)]} 0 1
[junit4] 2> 89606 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-3 (1463407846073827328)]} 0 0
[junit4] 2> 89610 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-3 (1463407846075924480)]} 0 0
[junit4] 2> 89611 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-3 (1463407846075924480)]} 0 3
[junit4] 2> 89611 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-3 (1463407846075924481)]} 0 1
[junit4] 2> 89611 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-3 (1463407846075924481)]} 0 3
[junit4] 2> 89666 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-4 (1463407846135693312)]} 0 1
[junit4] 2> 89666 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-4 (1463407846135693313)]} 0 1
[junit4] 2> 89670 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-4 (1463407846138839040)]} 0 1
[junit4] 2> 89670 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-4 (1463407846138839041)]} 0 0
[junit4] 2> 89670 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-4 (1463407846138839040)]} 0 2
[junit4] 2> 89671 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-4 (1463407846138839041)]} 0 3
[junit4] 2> 89707 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-5 (1463407846179733505)]} 0 0
[junit4] 2> 89708 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-5 (1463407846179733504)]} 0 1
[junit4] 2> 89712 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-5 (1463407846181830656)]} 0 1
[junit4] 2> 89712 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-5 (1463407846182879232)]} 0 1
[junit4] 2> 89712 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-5 (1463407846181830656)]} 0 3
[junit4] 2> 89713 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-5 (1463407846182879232)]} 0 3
[junit4] 2> 89714 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-2 (-1463407846187073536)]} 0 0
[junit4] 2> 89715 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-2 (-1463407846188122112)]} 0 0
[junit4] 2> 89718 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[2-2 (-1463407846190219264)]} 0 0
[junit4] 2> 89718 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[1-2 (-1463407846190219265)]} 0 0
[junit4] 2> 89719 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[2-2 (-1463407846190219264)]} 0 2
[junit4] 2> 89719 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[1-2 (-1463407846190219265)]} 0 2
[junit4] 2> 89721 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-6 (1463407846194413568)]} 0 0
[junit4] 2> 89721 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-6 (1463407846194413569)]} 0 0
[junit4] 2> 89725 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-6 (1463407846196510720)]} 0 0
[junit4] 2> 89725 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-6 (1463407846196510721)]} 0 0
[junit4] 2> 89726 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-6 (1463407846196510720)]} 0 3
[junit4] 2> 89726 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-6 (1463407846196510721)]} 0 3
[junit4] 2> 89742 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-7 (1463407846216433664)]} 0 0
[junit4] 2> 89743 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-7 (1463407846216433665)]} 0 1
[junit4] 2> 89746 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-7 (1463407846218530816)]} 0 0
[junit4] 2> 89747 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-7 (1463407846218530817)]} 0 1
[junit4] 2> 89747 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-7 (1463407846218530816)]} 0 3
[junit4] 2> 89747 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-7 (1463407846218530817)]} 0 3
[junit4] 2> 89794 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-8 (1463407846269911040)]} 0 1
[junit4] 2> 89794 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-8 (1463407846269911041)]} 0 1
[junit4] 2> 89798 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-8 (1463407846273056768)]} 0 1
[junit4] 2> 89798 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-8 (1463407846273056769)]} 0 0
[junit4] 2> 89799 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-8 (1463407846273056768)]} 0 3
[junit4] 2> 89799 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-8 (1463407846273056769)]} 0 3
[junit4] 2> 89858 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-9 (1463407846337019904)]} 0 1
[junit
[...truncated too long message...]
asc.Overseer.createOverseerNode ERROR Could not create Overseer node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 191141 T765 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@966553
[junit4] 2> 191156 T765 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=86,adds=86,deletesById=48,deletesByQuery=0,errors=0,cumulative_adds=86,cumulative_deletesById=48,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
[junit4] 2> 191156 T765 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 191156 T765 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 191157 T765 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_1,generation=1}
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_2,generation=2}
[junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4] 2> 191162 T765 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2;done=false>>]
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index;done=false>>]
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
[junit4] 2> 191164 T765 oasc.LeaderElector$1.process WARN java.lang.RuntimeException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1227)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
[junit4] 2> ... 14 more
[junit4] 2>
[junit4] 2> 191164 T765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 191164 T765 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 191165 T765 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 191165 T765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 191165 T765 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_GT, timezone=Asia/Brunei
[junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=80585760,total=235929600
[junit4] 2> NOTE: All tests run in this JVM: [DistribCursorPagingTest, ParsingFieldUpdateProcessorsTest, OverseerTest, RecoveryZkTest]
[junit4] Completed on J0 in 111.30s, 1 test, 1 failure <<< FAILURES!
[...truncated 1081 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908: There were test failures: 378 suites, 1598 tests, 1 failure, 37 ignored (16 assumptions)
Total time: 58 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build
# 9780 - Still Failing!
Posted by Mark Miller <ma...@gmail.com>.
Cause of this seems to be:
24422 T107 C8 P54679 oasc.SolrException.log ERROR SnapPull failed :org.apache.solr.common.SolrException: Index fetch failed :Â
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:506)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:324)
at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:446)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:234)
Caused by: java.io.IOException: MockDirectoryWrapper: file "_0.fdx" is still open for writing
at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:568)
at org.apache.solr.handler.SnapPuller.slowFileExists(SnapPuller.java:789)
at org.apache.solr.handler.SnapPuller.isIndexStale(SnapPuller.java:805)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:405)
... 4 more
Caused by: java.lang.RuntimeException: unclosed IndexOutput: _0.fdx
at org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:544)
at org.apache.lucene.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:516)
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:44)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:109)
at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:120)
at org.apache.lucene.index.StoredFieldsProcessor.initFieldsWriter(StoredFieldsProcessor.java:88)
at org.apache.lucene.index.StoredFieldsProcessor.finishDocument(StoredFieldsProcessor.java:120)
at org.apache.lucene.index.TwoStoredFieldsConsumers.finishDocument(TwoStoredFieldsConsumers.java:65)
at org.apache.lucene.index.DocFieldProcessor.finishDocument(DocFieldProcessor.java:273)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:271)
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:465)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1536)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:236)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:704)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:858)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:557)
at org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:100)
at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:96)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1957)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:785)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:429)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:218)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:134)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:953)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:744)
--Â
Mark Miller
about.me/markrmiller
On March 23, 2014 at 7:04:59 PM, Policeman Jenkins Server (jenkins@thetaphi.de) wrote:
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9780/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
There are still nodes recoverying - waited for 90 seconds
Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 90 seconds
at __randomizedtesting.SeedInfo.seed([92A83ABF6D073DEE:134EB4A71A585DD2]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:177)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:732)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1495)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:103)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Build Log:
[...truncated 10169 lines...]
[junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4] 2> 80886 T677 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 80887 T677 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /nju/sc
[junit4] 2> 80892 T677 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1395614468515
[junit4] 2> 80893 T677 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 80894 T678 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 80994 T677 oasc.ZkTestServer.run start zk server on port:46640
[junit4] 2> 80995 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81204 T684 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@afcd05 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81204 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81205 T677 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 81209 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81210 T686 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1598a49 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81210 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81211 T677 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 81213 T677 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 81215 T677 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 81216 T677 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 81218 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 81218 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 81220 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 81221 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 81224 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 81224 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 81226 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 81226 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 81228 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 81228 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 81230 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 81230 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 81231 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 81232 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 81233 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 81234 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 81235 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 81236 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 81237 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 81238 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 81239 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 81240 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 81475 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 81479 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38011
[junit4] 2> 81480 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864
[junit4] 2> 81482 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/'
[junit4] 2> 81536 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/solr.xml
[junit4] 2> 81599 T677 oasc.CoreContainer.<init> New CoreContainer 27496700
[junit4] 2> 81599 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/]
[junit4] 2> 81600 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 81603 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 81603 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 81604 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 81604 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 81604 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 81605 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 81605 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81607 T697 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1839970 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81607 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81609 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81609 T679 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144f11ac9a60002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 81611 T699 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6336bf name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81611 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81612 T677 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 81614 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 81616 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 81618 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 81620 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 81621 T677 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 81623 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81625 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81627 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 81628 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 81631 T677 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81631 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 81632 T677 oasc.Overseer.start Overseer (id=91462989808795651-127.0.0.1:38011_nju%2Fsc-n_0000000000) starting
[junit4] 2> 81637 T677 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 81645 T701 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 81645 T677 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 81646 T701 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 81647 T677 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 81648 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 81650 T700 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 81655 T702 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 81656 T702 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 81657 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 81657 T702 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 81658 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 81661 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 81662 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 82658 T702 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 82658 T702 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1
[junit4] 2> 82658 T702 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 82659 T702 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 82659 T702 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 82660 T702 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 82660 T702 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/'
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/.svn/' to classloader
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/classes/' to classloader
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/README' to classloader
[junit4] 2> 82714 T702 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 82762 T702 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 82763 T702 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 82788 T702 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 83255 T702 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 83258 T702 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 83260 T702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 83264 T702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 83285 T702 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 83286 T702 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/
[junit4] 2> 83289 T702 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 83292 T702 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data
[junit4] 2> 83292 T702 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index/
[junit4] 2> 83292 T702 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 83293 T702 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index
[junit4] 2> 83293 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
[junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 83299 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 83310 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 83313 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 83338 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 83339 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 83339 T702 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 83341 T702 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 83344 T702 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 83349 T702 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 83351 T702 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 83355 T702 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 83355 T702 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 83357 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
[junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 83358 T702 oass.SolrIndexSearcher.<init> Opening Searcher@1c23829[collection1] main
[junit4] 2> 83365 T703 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c23829[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 83368 T702 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 83369 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 83369 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 83369 T706 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38011/nju/sc collection:control_collection shard:shard1
[junit4] 2> 83370 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83371 T706 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 83372 T708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9657d4 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83372 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83373 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 83375 T677 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 83376 T706 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 83377 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C596 name=collection1 org.apache.solr.core.SolrCore@69814a url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc C596_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:38011/nju/sc, core=collection1, node_name=127.0.0.1:38011_nju%2Fsc}
[junit4] 2> 83378 T706 C596 P38011 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38011/nju/sc/collection1/
[junit4] 2> 83379 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncToMe http://127.0.0.1:38011/nju/sc/collection1/ has no replicas
[junit4] 2> 83379 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38011/nju/sc/collection1/ shard1
[junit4] 2> 83380 T706 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 83381 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83382 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83382 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83385 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83386 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83490 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83498 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83539 T706 oasc.ZkController.register We are http://127.0.0.1:38011/nju/sc/collection1/ and leader is http://127.0.0.1:38011/nju/sc/collection1/
[junit4] 2> 83539 T706 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38011/nju/sc
[junit4] 2> 83539 T706 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 83540 T706 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 83540 T706 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83542 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83542 T706 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83543 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83544 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 83546 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83631 T677 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 83631 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 83651 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39738
[junit4] 2> 83652 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998
[junit4] 2> 83655 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/'
[junit4] 2> 83655 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83669 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/solr.xml
[junit4] 2> 83720 T677 oasc.CoreContainer.<init> New CoreContainer 29042762
[junit4] 2> 83721 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/]
[junit4] 2> 83722 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 83725 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 83725 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 83726 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 83726 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 83726 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 83727 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 83727 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83729 T719 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5231ef name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83729 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83731 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83732 T721 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1075f95 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83733 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83737 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 84742 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39738_nju%2Fsc
[junit4] 2> 84743 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39738_nju%2Fsc
[junit4] 2> 84745 T708 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84745 T721 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84745 T699 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84751 T722 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 84752 T722 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84755 T722 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84756 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84757 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 84758 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 84759 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84862 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 84862 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 84862 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 85756 T722 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 85756 T722 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1
[junit4] 2> 85756 T722 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 85757 T722 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 85758 T722 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 85759 T722 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 85759 T722 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/'
[junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/.svn/' to classloader
[junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/classes/' to classloader
[junit4] 2> 85761 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/README' to classloader
[junit4] 2> 85807 T722 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 85855 T722 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 85856 T722 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 85866 T722 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 86312 T722 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 86315 T722 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 86317 T722 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 86321 T722 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 86342 T722 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 86343 T722 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/
[junit4] 2> 86343 T722 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 86346 T722 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1
[junit4] 2> 86346 T722 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index/
[junit4] 2> 86346 T722 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 86347 T722 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index
[junit4] 2> 86347 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
[junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 86353 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 86356 T722 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 86357 T722 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 86358 T722 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 86361 T722 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 86364 T722 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 86367 T722 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 86367 T722 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 86368 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
[junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 86369 T722 oass.SolrIndexSearcher.<init> Opening Searcher@383c54[collection1] main
[junit4] 2> 86376 T723 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@383c54[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 86381 T722 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 86382 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 86382 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 86382 T726 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39738/nju/sc collection:collection1 shard:shard1
[junit4] 2> 86383 T726 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 86387 T726 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C597 name=collection1 org.apache.solr.core.SolrCore@a6a2f3 url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc C597_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:39738/nju/sc, core=collection1, node_name=127.0.0.1:39738_nju%2Fsc}
[junit4] 2> 86389 T726 C597 P39738 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 86390 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86390 T726 C597 P39738 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 86391 T726 C597 P39738 oasc.SyncStrategy.syncToMe http://127.0.0.1:39738/nju/sc/collection1/ has no replicas
[junit4] 2> 86390 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86391 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39738/nju/sc/collection1/ shard1
[junit4] 2> 86391 T726 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 86392 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86394 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86394 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86394 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86397 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86399 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86501 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86501 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86501 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86547 T726 oasc.ZkController.register We are http://127.0.0.1:39738/nju/sc/collection1/ and leader is http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 86547 T726 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39738/nju/sc
[junit4] 2> 86547 T726 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 86548 T726 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 86548 T726 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86550 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86550 T726 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86551 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86551 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 86553 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86655 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86655 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86655 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86667 T677 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 86668 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 86670 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40129
[junit4] 2> 86671 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 86671 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 86672 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005
[junit4] 2> 86672 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/'
[junit4] 2> 86704 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/solr.xml
[junit4] 2> 86751 T677 oasc.CoreContainer.<init> New CoreContainer 10907151
[junit4] 2> 86751 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/]
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 86756 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 86756 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 86756 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 86757 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 86757 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 86758 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 86759 T737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@83c633 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 86760 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 86762 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 86763 T739 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18288ea name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 86763 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 86768 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 87770 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40129_nju%2Fsc
[junit4] 2> 87771 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40129_nju%2Fsc
[junit4] 2> 87773 T699 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87773 T708 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87773 T721 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87774 T739 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87781 T740 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 87781 T740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 87783 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87783 T740 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 87785 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 87789 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87890 T739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 88785 T740 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 88785 T740 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1
[junit4] 2> 88785 T740 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 88786 T740 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 88786 T740 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 88787 T740 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 88787 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/'
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/.svn/' to classloader
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/classes/' to classloader
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/README' to classloader
[junit4] 2> 88833 T740 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 88882 T740 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 88883 T740 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 88895 T740 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 89367 T740 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 89370 T740 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 89372 T740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 89375 T740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 89392 T740 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 89393 T740 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/
[junit4] 2> 89393 T740 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 89394 T740 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
[junit4] 2> 89395 T740 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index/
[junit4] 2> 89395 T740 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 89395 T740 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
[junit4] 2> 89396 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
[junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 89404 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 89404 T740 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 89405 T740 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 89408 T740 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 89411 T740 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 89414 T740 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 89414 T740 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 89415 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 89415 T740 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
[junit4] 2> 89416 T740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 89416 T740 oass.SolrIndexSearcher.<init> Opening Searcher@c599fa[collection1] main
[junit4] 2> 89422 T741 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c599fa[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 89426 T740 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 89426 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 89427 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 89426 T744 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40129/nju/sc collection:collection1 shard:shard1
[junit4] 2> 89428 T677 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89430 T744 oasc.ZkController.register We are http://127.0.0.1:40129/nju/sc/collection1/ and leader is http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 89430 T744 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40129/nju/sc
[junit4] 2> 89431 T744 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 89431 T744 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> 89431 T744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C598 name=collection1 org.apache.solr.core.SolrCore@1ec0a26 url=http://127.0.0.1:40129/nju/sc/collection1 node=127.0.0.1:40129_nju%2Fsc C598_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:40129/nju/sc, core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
[junit4] 2> 89432 T747 C598 P40129 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 89433 T747 C598 P40129 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
[junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89436 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89437 T714 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 89437 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 89438 T714 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89439 T714 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 89439 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89440 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> ASYNC NEW_CORE C599 name=collection1 org.apache.solr.core.SolrCore@69814a url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc C599_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:38011/nju/sc, core=collection1, node_name=127.0.0.1:38011_nju%2Fsc, leader=true}
[junit4] 2> 89447 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-0 (1463407845892423680)]} 0 14
[junit4] 2> 89448 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-0 (1463407845892423681)]} 0 15
[junit4] 2> C598_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=http://127.0.0.1:40129/nju/sc, core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
[junit4] 2> 89458 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-0 (1463407845911298048)]} 0 2
[junit4] 2> 89458 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-0 (1463407845911298049)]} 0 2
[junit4] 2> ASYNC NEW_CORE C600 name=collection1 org.apache.solr.core.SolrCore@a6a2f3 url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc C600_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:39738/nju/sc, core=collection1, node_name=127.0.0.1:39738_nju%2Fsc, leader=true}
[junit4] 2> 89459 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-0 (1463407845911298048)]} 0 8
[junit4] 2> 89460 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-0 (1463407845911298049)]} 0 9
[junit4] 2> 89527 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-0 (-1463407845989941249)]} 0 1
[junit4] 2> 89527 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-0 (-1463407845989941248)]} 0 1
[junit4] 2> 89532 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[2-0 (-1463407845994135553)]} 0 0
[junit4] 2> 89532 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[1-0 (-1463407845994135552)]} 0 0
[junit4] 2> 89533 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[2-0 (-1463407845994135553)]} 0 3
[junit4] 2> 89533 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[1-0 (-1463407845994135552)]} 0 3
[junit4] 2> 89536 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-1 (1463407845999378432)]} 0 1
[junit4] 2> 89536 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-1 (1463407845999378433)]} 0 1
[junit4] 2> 89540 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-1 (1463407846002524160)]} 0 1
[junit4] 2> 89540 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-1 (1463407846002524160)]} 0 2
[junit4] 2> 89541 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-1 (1463407846002524161)]} 0 1
[junit4] 2> 89541 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-1 (1463407846002524161)]} 0 3
[junit4] 2> 89589 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-2 (1463407846054952960)]} 0 1
[junit4] 2> 89589 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-2 (1463407846054952961)]} 0 1
[junit4] 2> 89593 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-2 (1463407846058098688)]} 0 0
[junit4] 2> 89594 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-2 (1463407846058098688)]} 0 3
[junit4] 2> 89594 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-2 (1463407846058098689)]} 0 1
[junit4] 2> 89595 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-2 (1463407846058098689)]} 0 4
[junit4] 2> 89606 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-3 (1463407846072778752)]} 0 1
[junit4] 2> 89606 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-3 (1463407846073827328)]} 0 0
[junit4] 2> 89610 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-3 (1463407846075924480)]} 0 0
[junit4] 2> 89611 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-3 (1463407846075924480)]} 0 3
[junit4] 2> 89611 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-3 (1463407846075924481)]} 0 1
[junit4] 2> 89611 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-3 (1463407846075924481)]} 0 3
[junit4] 2> 89666 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-4 (1463407846135693312)]} 0 1
[junit4] 2> 89666 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-4 (1463407846135693313)]} 0 1
[junit4] 2> 89670 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-4 (1463407846138839040)]} 0 1
[junit4] 2> 89670 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-4 (1463407846138839041)]} 0 0
[junit4] 2> 89670 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-4 (1463407846138839040)]} 0 2
[junit4] 2> 89671 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-4 (1463407846138839041)]} 0 3
[junit4] 2> 89707 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-5 (1463407846179733505)]} 0 0
[junit4] 2> 89708 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-5 (1463407846179733504)]} 0 1
[junit4] 2> 89712 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-5 (1463407846181830656)]} 0 1
[junit4] 2> 89712 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-5 (1463407846182879232)]} 0 1
[junit4] 2> 89712 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-5 (1463407846181830656)]} 0 3
[junit4] 2> 89713 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-5 (1463407846182879232)]} 0 3
[junit4] 2> 89714 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-2 (-1463407846187073536)]} 0 0
[junit4] 2> 89715 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-2 (-1463407846188122112)]} 0 0
[junit4] 2> 89718 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[2-2 (-1463407846190219264)]} 0 0
[junit4] 2> 89718 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[1-2 (-1463407846190219265)]} 0 0
[junit4] 2> 89719 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[2-2 (-1463407846190219264)]} 0 2
[junit4] 2> 89719 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[1-2 (-1463407846190219265)]} 0 2
[junit4] 2> 89721 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-6 (1463407846194413568)]} 0 0
[junit4] 2> 89721 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-6 (1463407846194413569)]} 0 0
[junit4] 2> 89725 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-6 (1463407846196510720)]} 0 0
[junit4] 2> 89725 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-6 (1463407846196510721)]} 0 0
[junit4] 2> 89726 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-6 (1463407846196510720)]} 0 3
[junit4] 2> 89726 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-6 (1463407846196510721)]} 0 3
[junit4] 2> 89742 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-7 (1463407846216433664)]} 0 0
[junit4] 2> 89743 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-7 (1463407846216433665)]} 0 1
[junit4] 2> 89746 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-7 (1463407846218530816)]} 0 0
[junit4] 2> 89747 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-7 (1463407846218530817)]} 0 1
[junit4] 2> 89747 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-7 (1463407846218530816)]} 0 3
[junit4] 2> 89747 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-7 (1463407846218530817)]} 0 3
[junit4] 2> 89794 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-8 (1463407846269911040)]} 0 1
[junit4] 2> 89794 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-8 (1463407846269911041)]} 0 1
[junit4] 2> 89798 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-8 (1463407846273056768)]} 0 1
[junit4] 2> 89798 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-8 (1463407846273056769)]} 0 0
[junit4] 2> 89799 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-8 (1463407846273056768)]} 0 3
[junit4] 2> 89799 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-8 (1463407846273056769)]} 0 3
[junit4] 2> 89858 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-9 (1463407846337019904)]} 0 1
[junit
[...truncated too long message...]
asc.Overseer.createOverseerNode ERROR Could not create Overseer node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 191141 T765 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@966553
[junit4] 2> 191156 T765 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=86,adds=86,deletesById=48,deletesByQuery=0,errors=0,cumulative_adds=86,cumulative_deletesById=48,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
[junit4] 2> 191156 T765 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 191156 T765 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 191157 T765 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_1,generation=1}
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_2,generation=2}
[junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4] 2> 191162 T765 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2;done=false>>]
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index;done=false>>]
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
[junit4] 2> 191164 T765 oasc.LeaderElector$1.process WARN java.lang.RuntimeException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1227)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
[junit4] 2> ... 14 more
[junit4] 2>
[junit4] 2> 191164 T765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 191164 T765 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 191165 T765 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 191165 T765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 191165 T765 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_GT, timezone=Asia/Brunei
[junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=80585760,total=235929600
[junit4] 2> NOTE: All tests run in this JVM: [DistribCursorPagingTest, ParsingFieldUpdateProcessorsTest, OverseerTest, RecoveryZkTest]
[junit4] Completed on J0 in 111.30s, 1 test, 1 failure <<< FAILURES!
[...truncated 1081 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908: There were test failures: 378 suites, 1598 tests, 1 failure, 37 ignored (16 assumptions)
Total time: 58 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.7.0_51) - Build # 9780
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/9780/
Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
Error Message:
There are still nodes recoverying - waited for 90 seconds
Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 90 seconds
at __randomizedtesting.SeedInfo.seed([92A83ABF6D073DEE:134EB4A71A585DD2]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:177)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:732)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1495)
at org.apache.solr.cloud.RecoveryZkTest.doTest(RecoveryZkTest.java:103)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:744)
Build Log:
[...truncated 10169 lines...]
[junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4] 2> 80886 T677 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
[junit4] 2> 80887 T677 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /nju/sc
[junit4] 2> 80892 T677 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1395614468515
[junit4] 2> 80893 T677 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 80894 T678 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 80994 T677 oasc.ZkTestServer.run start zk server on port:46640
[junit4] 2> 80995 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81204 T684 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@afcd05 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81204 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81205 T677 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 81209 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81210 T686 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1598a49 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81210 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81211 T677 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 81213 T677 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 81215 T677 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 81216 T677 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 81218 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 81218 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 81220 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 81221 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 81224 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 81224 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 81226 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 81226 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 81228 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 81228 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 81230 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 81230 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 81231 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 81232 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 81233 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 81234 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 81235 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 81236 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 81237 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 81238 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 81239 T677 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 81240 T677 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 81475 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 81479 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38011
[junit4] 2> 81480 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 81481 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864
[junit4] 2> 81482 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/'
[junit4] 2> 81536 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/solr.xml
[junit4] 2> 81599 T677 oasc.CoreContainer.<init> New CoreContainer 27496700
[junit4] 2> 81599 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/]
[junit4] 2> 81600 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 81601 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 81602 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 81603 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 81603 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 81604 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 81604 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 81604 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 81605 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 81605 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81607 T697 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1839970 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81607 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81609 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 81609 T679 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x144f11ac9a60002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:744)
[junit4] 2>
[junit4] 2> 81611 T699 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6336bf name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 81611 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 81612 T677 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 81614 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 81616 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 81618 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 81620 T677 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 81621 T677 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 81623 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81625 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81627 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 81628 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 81631 T677 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:38011_nju%2Fsc
[junit4] 2> 81631 T677 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 81632 T677 oasc.Overseer.start Overseer (id=91462989808795651-127.0.0.1:38011_nju%2Fsc-n_0000000000) starting
[junit4] 2> 81637 T677 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 81645 T701 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 81645 T677 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 81646 T701 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 81647 T677 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 81648 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 81650 T700 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 81655 T702 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 81656 T702 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 81657 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 81657 T702 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 81658 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 81659 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 81661 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 81662 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 82658 T702 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 82658 T702 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1
[junit4] 2> 82658 T702 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 82659 T702 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 82659 T702 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 82660 T702 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 82660 T702 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/'
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/.svn/' to classloader
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/classes/' to classloader
[junit4] 2> 82661 T702 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/lib/README' to classloader
[junit4] 2> 82714 T702 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 82762 T702 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 82763 T702 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 82788 T702 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 83255 T702 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 83258 T702 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 83260 T702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 83264 T702 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 83285 T702 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 83286 T702 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-controljetty-1395614468864/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/
[junit4] 2> 83289 T702 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 83292 T702 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data
[junit4] 2> 83292 T702 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index/
[junit4] 2> 83292 T702 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 83293 T702 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/control/data/index
[junit4] 2> 83293 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
[junit4] 2> 83294 T702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 83299 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 83310 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 83313 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 83314 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 83338 T702 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 83339 T702 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 83339 T702 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 83341 T702 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 83342 T702 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 83343 T702 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 83344 T702 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 83349 T702 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 83351 T702 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 83355 T702 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 83355 T702 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 83357 T702 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@1d8c25d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16cdb03),segFN=segments_1,generation=1}
[junit4] 2> 83357 T702 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 83358 T702 oass.SolrIndexSearcher.<init> Opening Searcher@1c23829[collection1] main
[junit4] 2> 83365 T703 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c23829[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 83368 T702 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 83369 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 83369 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 83369 T706 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38011/nju/sc collection:control_collection shard:shard1
[junit4] 2> 83370 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83371 T706 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 83372 T708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9657d4 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83372 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83373 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 83375 T677 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 83376 T706 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 83377 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 83378 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C596 name=collection1 org.apache.solr.core.SolrCore@69814a url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc C596_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:38011/nju/sc, core=collection1, node_name=127.0.0.1:38011_nju%2Fsc}
[junit4] 2> 83378 T706 C596 P38011 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38011/nju/sc/collection1/
[junit4] 2> 83379 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 83379 T706 C596 P38011 oasc.SyncStrategy.syncToMe http://127.0.0.1:38011/nju/sc/collection1/ has no replicas
[junit4] 2> 83379 T706 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38011/nju/sc/collection1/ shard1
[junit4] 2> 83380 T706 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 83381 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83382 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83382 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83385 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83386 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83490 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83498 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83539 T706 oasc.ZkController.register We are http://127.0.0.1:38011/nju/sc/collection1/ and leader is http://127.0.0.1:38011/nju/sc/collection1/
[junit4] 2> 83539 T706 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38011/nju/sc
[junit4] 2> 83539 T706 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 83540 T706 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 83540 T706 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83541 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83542 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83542 T706 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83543 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 83544 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:38011/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:38011_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 83546 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 83631 T677 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 83631 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 83651 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39738
[junit4] 2> 83652 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998
[junit4] 2> 83655 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83655 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/'
[junit4] 2> 83655 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 83669 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/solr.xml
[junit4] 2> 83720 T677 oasc.CoreContainer.<init> New CoreContainer 29042762
[junit4] 2> 83721 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/]
[junit4] 2> 83722 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 83723 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 83724 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 83725 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 83725 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 83726 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 83726 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 83726 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 83727 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 83727 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83729 T719 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5231ef name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83729 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83731 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 83732 T721 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1075f95 name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 83733 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 83737 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 84742 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39738_nju%2Fsc
[junit4] 2> 84743 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39738_nju%2Fsc
[junit4] 2> 84745 T708 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84745 T721 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84745 T699 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 84751 T722 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 84752 T722 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84755 T722 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 84755 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84756 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84757 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 84757 T700 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 84758 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 84759 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 84862 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 84862 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 84862 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 85756 T722 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 85756 T722 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1
[junit4] 2> 85756 T722 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 85757 T722 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 85758 T722 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 85759 T722 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 85759 T722 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/'
[junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/.svn/' to classloader
[junit4] 2> 85760 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/classes/' to classloader
[junit4] 2> 85761 T722 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/lib/README' to classloader
[junit4] 2> 85807 T722 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 85855 T722 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 85856 T722 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 85866 T722 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 86312 T722 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 86315 T722 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 86317 T722 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 86321 T722 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 86342 T722 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 86343 T722 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1395614470998/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/
[junit4] 2> 86343 T722 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 86346 T722 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1
[junit4] 2> 86346 T722 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index/
[junit4] 2> 86346 T722 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 86347 T722 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty1/index
[junit4] 2> 86347 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
[junit4] 2> 86348 T722 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 86353 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 86354 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 86355 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 86356 T722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 86356 T722 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 86357 T722 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 86358 T722 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 86359 T722 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 86360 T722 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 86361 T722 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 86364 T722 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 86367 T722 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 86367 T722 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 86368 T722 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@69864f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8423ec),segFN=segments_1,generation=1}
[junit4] 2> 86369 T722 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 86369 T722 oass.SolrIndexSearcher.<init> Opening Searcher@383c54[collection1] main
[junit4] 2> 86376 T723 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@383c54[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 86381 T722 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 86382 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 86382 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 86382 T726 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39738/nju/sc collection:collection1 shard:shard1
[junit4] 2> 86383 T726 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 86387 T726 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 86389 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 86389 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C597 name=collection1 org.apache.solr.core.SolrCore@a6a2f3 url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc C597_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:39738/nju/sc, core=collection1, node_name=127.0.0.1:39738_nju%2Fsc}
[junit4] 2> 86389 T726 C597 P39738 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 86390 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86390 T726 C597 P39738 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 86391 T726 C597 P39738 oasc.SyncStrategy.syncToMe http://127.0.0.1:39738/nju/sc/collection1/ has no replicas
[junit4] 2> 86390 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86391 T726 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39738/nju/sc/collection1/ shard1
[junit4] 2> 86391 T726 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 86392 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86394 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86394 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86394 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86397 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86399 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86501 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86501 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86501 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86547 T726 oasc.ZkController.register We are http://127.0.0.1:39738/nju/sc/collection1/ and leader is http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 86547 T726 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39738/nju/sc
[junit4] 2> 86547 T726 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 86548 T726 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 86548 T726 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86549 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86550 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86550 T726 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86551 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 86551 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:39738/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39738_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 86553 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 86655 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86655 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86655 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 86667 T677 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 86668 T677 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 86670 T677 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40129
[junit4] 2> 86671 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 86671 T677 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 86672 T677 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005
[junit4] 2> 86672 T677 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/'
[junit4] 2> 86704 T677 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/solr.xml
[junit4] 2> 86751 T677 oasc.CoreContainer.<init> New CoreContainer 10907151
[junit4] 2> 86751 T677 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/]
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 86753 T677 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 86754 T677 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 86755 T677 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 86756 T677 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 86756 T677 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 86756 T677 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 86757 T677 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46640/solr
[junit4] 2> 86757 T677 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 86758 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 86759 T737 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@83c633 name:ZooKeeperConnection Watcher:127.0.0.1:46640 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 86760 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 86762 T677 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 86763 T739 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18288ea name:ZooKeeperConnection Watcher:127.0.0.1:46640/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 86763 T677 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 86768 T677 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 87770 T677 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40129_nju%2Fsc
[junit4] 2> 87771 T677 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40129_nju%2Fsc
[junit4] 2> 87773 T699 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87773 T708 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87773 T721 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87774 T739 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 87781 T740 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 87781 T740 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 87783 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87784 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87783 T740 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 87785 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 87786 T700 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 87789 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 87890 T739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 87890 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 88785 T740 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 88785 T740 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1
[junit4] 2> 88785 T740 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 88786 T740 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 88786 T740 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 88787 T740 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 88787 T740 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/'
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/.svn/' to classloader
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/classes/' to classloader
[junit4] 2> 88788 T740 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/lib/README' to classloader
[junit4] 2> 88833 T740 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 88882 T740 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 88883 T740 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 88895 T740 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 89367 T740 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 89370 T740 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 89372 T740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 89375 T740 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 89392 T740 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 89393 T740 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.RecoveryZkTest-jetty2-1395614474005/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/
[junit4] 2> 89393 T740 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1034977
[junit4] 2> 89394 T740 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
[junit4] 2> 89395 T740 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index/
[junit4] 2> 89395 T740 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 89395 T740 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
[junit4] 2> 89396 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
[junit4] 2> 89397 T740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 89402 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 89403 T740 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 89404 T740 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 89404 T740 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 89405 T740 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 89406 T740 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 89407 T740 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 89408 T740 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 89411 T740 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 89414 T740 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 89414 T740 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 89415 T740 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 89415 T740 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@11a45d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@16d389f),segFN=segments_1,generation=1}
[junit4] 2> 89416 T740 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 89416 T740 oass.SolrIndexSearcher.<init> Opening Searcher@c599fa[collection1] main
[junit4] 2> 89422 T741 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c599fa[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 89426 T740 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 89426 T677 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4] 2> 89427 T677 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 89426 T744 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40129/nju/sc collection:collection1 shard:shard1
[junit4] 2> 89428 T677 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89430 T744 oasc.ZkController.register We are http://127.0.0.1:40129/nju/sc/collection1/ and leader is http://127.0.0.1:39738/nju/sc/collection1/
[junit4] 2> 89430 T744 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40129/nju/sc
[junit4] 2> 89431 T744 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 89431 T744 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> 89431 T744 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C598 name=collection1 org.apache.solr.core.SolrCore@1ec0a26 url=http://127.0.0.1:40129/nju/sc/collection1 node=127.0.0.1:40129_nju%2Fsc C598_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:40129/nju/sc, core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
[junit4] 2> 89432 T747 C598 P40129 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 89433 T747 C598 P40129 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish publishing core=collection1 state=recovering collection=collection1
[junit4] 2> 89434 T747 C598 P40129 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89435 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89436 T700 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89437 T714 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 89437 T700 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "base_url":"http://127.0.0.1:40129/nju/sc",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:40129_nju%2Fsc",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 89438 T714 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 89439 T714 oasha.CoreAdminHandler.handleWaitForStateAction Will wait a max of 183 seconds to see collection1 (shard1 of collection1) have state: recovering
[junit4] 2> 89439 T699 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 89440 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T739 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T708 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 89440 T699 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> ASYNC NEW_CORE C599 name=collection1 org.apache.solr.core.SolrCore@69814a url=http://127.0.0.1:38011/nju/sc/collection1 node=127.0.0.1:38011_nju%2Fsc C599_STATE=coll:control_collection core:collection1 props:{state=active, base_url=http://127.0.0.1:38011/nju/sc, core=collection1, node_name=127.0.0.1:38011_nju%2Fsc, leader=true}
[junit4] 2> 89447 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-0 (1463407845892423680)]} 0 14
[junit4] 2> 89448 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-0 (1463407845892423681)]} 0 15
[junit4] 2> C598_STATE=coll:collection1 core:collection1 props:{state=recovering, base_url=http://127.0.0.1:40129/nju/sc, core=collection1, node_name=127.0.0.1:40129_nju%2Fsc}
[junit4] 2> 89458 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-0 (1463407845911298048)]} 0 2
[junit4] 2> 89458 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-0 (1463407845911298049)]} 0 2
[junit4] 2> ASYNC NEW_CORE C600 name=collection1 org.apache.solr.core.SolrCore@a6a2f3 url=http://127.0.0.1:39738/nju/sc/collection1 node=127.0.0.1:39738_nju%2Fsc C600_STATE=coll:collection1 core:collection1 props:{state=active, base_url=http://127.0.0.1:39738/nju/sc, core=collection1, node_name=127.0.0.1:39738_nju%2Fsc, leader=true}
[junit4] 2> 89459 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-0 (1463407845911298048)]} 0 8
[junit4] 2> 89460 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-0 (1463407845911298049)]} 0 9
[junit4] 2> 89527 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-0 (-1463407845989941249)]} 0 1
[junit4] 2> 89527 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-0 (-1463407845989941248)]} 0 1
[junit4] 2> 89532 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[2-0 (-1463407845994135553)]} 0 0
[junit4] 2> 89532 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[1-0 (-1463407845994135552)]} 0 0
[junit4] 2> 89533 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[2-0 (-1463407845994135553)]} 0 3
[junit4] 2> 89533 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[1-0 (-1463407845994135552)]} 0 3
[junit4] 2> 89536 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-1 (1463407845999378432)]} 0 1
[junit4] 2> 89536 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-1 (1463407845999378433)]} 0 1
[junit4] 2> 89540 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-1 (1463407846002524160)]} 0 1
[junit4] 2> 89540 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-1 (1463407846002524160)]} 0 2
[junit4] 2> 89541 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-1 (1463407846002524161)]} 0 1
[junit4] 2> 89541 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-1 (1463407846002524161)]} 0 3
[junit4] 2> 89589 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-2 (1463407846054952960)]} 0 1
[junit4] 2> 89589 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-2 (1463407846054952961)]} 0 1
[junit4] 2> 89593 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-2 (1463407846058098688)]} 0 0
[junit4] 2> 89594 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-2 (1463407846058098688)]} 0 3
[junit4] 2> 89594 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-2 (1463407846058098689)]} 0 1
[junit4] 2> 89595 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-2 (1463407846058098689)]} 0 4
[junit4] 2> 89606 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-3 (1463407846072778752)]} 0 1
[junit4] 2> 89606 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-3 (1463407846073827328)]} 0 0
[junit4] 2> 89610 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-3 (1463407846075924480)]} 0 0
[junit4] 2> 89611 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-3 (1463407846075924480)]} 0 3
[junit4] 2> 89611 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-3 (1463407846075924481)]} 0 1
[junit4] 2> 89611 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-3 (1463407846075924481)]} 0 3
[junit4] 2> 89666 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-4 (1463407846135693312)]} 0 1
[junit4] 2> 89666 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-4 (1463407846135693313)]} 0 1
[junit4] 2> 89670 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-4 (1463407846138839040)]} 0 1
[junit4] 2> 89670 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-4 (1463407846138839041)]} 0 0
[junit4] 2> 89670 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-4 (1463407846138839040)]} 0 2
[junit4] 2> 89671 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-4 (1463407846138839041)]} 0 3
[junit4] 2> 89707 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-5 (1463407846179733505)]} 0 0
[junit4] 2> 89708 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-5 (1463407846179733504)]} 0 1
[junit4] 2> 89712 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-5 (1463407846181830656)]} 0 1
[junit4] 2> 89712 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-5 (1463407846182879232)]} 0 1
[junit4] 2> 89712 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-5 (1463407846181830656)]} 0 3
[junit4] 2> 89713 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-5 (1463407846182879232)]} 0 3
[junit4] 2> 89714 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[2-2 (-1463407846187073536)]} 0 0
[junit4] 2> 89715 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {delete=[1-2 (-1463407846188122112)]} 0 0
[junit4] 2> 89718 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[2-2 (-1463407846190219264)]} 0 0
[junit4] 2> 89718 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {delete=[1-2 (-1463407846190219265)]} 0 0
[junit4] 2> 89719 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[2-2 (-1463407846190219264)]} 0 2
[junit4] 2> 89719 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {delete=[1-2 (-1463407846190219265)]} 0 2
[junit4] 2> 89721 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-6 (1463407846194413568)]} 0 0
[junit4] 2> 89721 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-6 (1463407846194413569)]} 0 0
[junit4] 2> 89725 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-6 (1463407846196510720)]} 0 0
[junit4] 2> 89725 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-6 (1463407846196510721)]} 0 0
[junit4] 2> 89726 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-6 (1463407846196510720)]} 0 3
[junit4] 2> 89726 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-6 (1463407846196510721)]} 0 3
[junit4] 2> 89742 T693 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-7 (1463407846216433664)]} 0 0
[junit4] 2> 89743 T695 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-7 (1463407846216433665)]} 0 1
[junit4] 2> 89746 T730 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-7 (1463407846218530816)]} 0 0
[junit4] 2> 89747 T735 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-7 (1463407846218530817)]} 0 1
[junit4] 2> 89747 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-7 (1463407846218530816)]} 0 3
[junit4] 2> 89747 T717 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-7 (1463407846218530817)]} 0 3
[junit4] 2> 89794 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[2-8 (1463407846269911040)]} 0 1
[junit4] 2> 89794 T694 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-8 (1463407846269911041)]} 0 1
[junit4] 2> 89798 T733 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[2-8 (1463407846273056768)]} 0 1
[junit4] 2> 89798 T734 C598 P40129 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={distrib.from=http://127.0.0.1:39738/nju/sc/collection1/&version=2&wt=javabin&update.distrib=FROMLEADER} {add=[1-8 (1463407846273056769)]} 0 0
[junit4] 2> 89799 T715 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[2-8 (1463407846273056768)]} 0 3
[junit4] 2> 89799 T716 C600 P39738 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin} {add=[1-8 (1463407846273056769)]} 0 3
[junit4] 2> 89858 T690 C599 P38011 oasup.LogUpdateProcessor.finish [collection1] webapp=/nju/sc path=/update params={version=2&wt=javabin&CONTROL=TRUE} {add=[1-9 (1463407846337019904)]} 0 1
[junit
[...truncated too long message...]
asc.Overseer.createOverseerNode ERROR Could not create Overseer node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 191141 T765 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@966553
[junit4] 2> 191156 T765 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=86,adds=86,deletesById=48,deletesByQuery=0,errors=0,cumulative_adds=86,cumulative_deletesById=48,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
[junit4] 2> 191156 T765 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 191156 T765 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 191157 T765 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_1,generation=1}
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7ed6f6 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c64f98),segFN=segments_2,generation=2}
[junit4] 2> 191161 T765 C612 P40129 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4] 2> 191162 T765 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2;done=false>>]
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index;done=false>>]
[junit4] 2> 191163 T765 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1395614468515/jetty2/index
[junit4] 2> 191164 T765 oasc.LeaderElector$1.process WARN java.lang.RuntimeException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1227)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1178)
[junit4] 2> at org.apache.solr.cloud.Overseer.getInQueue(Overseer.java:1174)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:191)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:289)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:422)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:212)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:311)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$9.execute(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:308)
[junit4] 2> at org.apache.solr.cloud.Overseer.createOverseerNode(Overseer.java:1218)
[junit4] 2> ... 14 more
[junit4] 2>
[junit4] 2> 191164 T765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 191164 T765 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 191165 T765 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 191165 T765 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 191165 T765 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=es_GT, timezone=Asia/Brunei
[junit4] 2> NOTE: Linux 3.8.0-37-generic i386/Oracle Corporation 1.7.0_51 (32-bit)/cpus=8,threads=1,free=80585760,total=235929600
[junit4] 2> NOTE: All tests run in this JVM: [DistribCursorPagingTest, ParsingFieldUpdateProcessorsTest, OverseerTest, RecoveryZkTest]
[junit4] Completed on J0 in 111.30s, 1 test, 1 failure <<< FAILURES!
[...truncated 1081 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:490: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1276: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:908: There were test failures: 378 suites, 1598 tests, 1 failure, 37 ignored (16 assumptions)
Total time: 58 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_51 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure