You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/09/11 14:10:53 UTC
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 170 -
Failure!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/170/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
All tests passed
Build Log:
[...truncated 9398 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/build.xml:826: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/build.xml:770: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/build.xml:495: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/common-build.xml:2256: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/module-build.xml:58: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/module-build.xml:55: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/common-build.xml:831: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/common-build.xml:843: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/common-build.xml:2044: Compile failed; see the compiler error output for details.
Total time: 16 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 174 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/174/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
2 tests failed.
FAILED: org.apache.solr.core.TestLazyCores.testCachingLimit
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([29196813FF046BA8]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.core.TestLazyCores
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([29196813FF046BA8]:0)
Build Log:
[...truncated 13718 lines...]
[junit4] Suite: org.apache.solr.core.TestLazyCores
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/init-core-data-001
[junit4] 2> 2397775 WARN (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=39 numCloses=39
[junit4] 2> 2397775 INFO (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 2397777 INFO (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 2397777 INFO (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 2397779 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testBadConfigsGenerateErrors
[junit4] 2> 2397791 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2397834 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2397843 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2397843 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2397847 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.c.CorePropertiesLocator Found 6 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001
[junit4] 2> 2397847 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.c.CorePropertiesLocator Cores are: [badConfig2, badSchema1, core2, badConfig1, badSchema2, core1]
[junit4] 2> 2397847 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2397870 INFO (coreLoadExecutor-7936-thread-2) [ x:badSchema1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2397870 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2397872 INFO (coreLoadExecutor-7936-thread-1) [ x:badConfig2] o.a.s.c.Config Ignoring exception matching \QSAXParseException\E
[junit4] 2> 2397876 INFO (coreLoadExecutor-7936-thread-2) [ x:badSchema1] o.a.s.c.Config Ignoring exception matching \QSAXParseException\E
[junit4] 2> 2397877 INFO (coreLoadExecutor-7936-thread-1) [ x:badConfig1] o.a.s.c.Config Ignoring exception matching \QSAXParseException\E
[junit4] 2> 2397877 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.s.IndexSchema [core2] Schema name=tiny
[junit4] 2> 2397879 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2397879 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.c.CoreContainer Creating SolrCore 'core2' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/core2, trusted=true
[junit4] 2> 2397880 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.core2' (registry 'solr.core.core2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2397880 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2397880 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.c.SolrCore [[core2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/core2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/core2/data/]
[junit4] 2> 2397882 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2397885 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2397885 INFO (coreLoadExecutor-7936-thread-2) [ x:badSchema2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2397900 INFO (coreLoadExecutor-7936-thread-2) [ x:badSchema2] o.a.s.c.Config Ignoring exception matching \QSAXParseException\E
[junit4] 2> 2397903 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.s.IndexSchema [core1] Schema name=tiny
[junit4] 2> 2397905 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2397905 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.c.CoreContainer Creating SolrCore 'core1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/core1, trusted=true
[junit4] 2> 2397905 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.core1' (registry 'solr.core.core1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2397905 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2397905 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.c.SolrCore [[core1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/core1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/core1/data/]
[junit4] 2> 2397908 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2397960 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2397960 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2397961 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1649012097, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2397962 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.s.SolrIndexSearcher Opening [Searcher@47a82bfc[core2] main]
[junit4] 2> 2397962 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/core2/conf
[junit4] 2> 2397963 INFO (coreLoadExecutor-7936-thread-3) [ x:core2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2397963 INFO (searcherExecutor-7937-thread-1-processing-x:core2) [ x:core2] o.a.s.c.SolrCore [core2] Registered new searcher Searcher@47a82bfc[core2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2397964 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2397964 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2397966 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1649012097, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2397966 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.s.SolrIndexSearcher Opening [Searcher@44d18fb5[core1] main]
[junit4] 2> 2397966 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/core1/conf
[junit4] 2> 2397967 INFO (coreLoadExecutor-7936-thread-1) [ x:core1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2397968 INFO (searcherExecutor-7938-thread-1-processing-x:core1) [ x:core1] o.a.s.c.SolrCore [core1] Registered new searcher Searcher@44d18fb5[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2397969 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 2397969 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@44bbfa95 commitCommandVersion:0
[junit4] 2> 2397979 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@29d5e0a[core1] main]
[junit4] 2> 2397980 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 2397981 INFO (searcherExecutor-7938-thread-1) [ ] o.a.s.c.SolrCore [core1] Registered new searcher Searcher@29d5e0a[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c7)))}
[junit4] 2> 2397986 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.c.S.Request [core1] webapp=null path=null params={q={!prefix+f%3Dv_t}hel&wt=xml} hits=2 status=0 QTime=0
[junit4] 2> 2397989 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.c.S.Request [core1] webapp=null path=null params={q={!raw+f%3Dv_t}hello&wt=xml} hits=2 status=0 QTime=0
[junit4] 2> 2397989 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.c.S.Request [core1] webapp=null path=null params={q={!raw+f%3Dv_t}Hello&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2397990 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ ] o.a.s.c.S.Request [core1] webapp=null path=null params={q={!raw+f%3Dv_f}1.5&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2398004 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398009 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.IndexSchema [badConfig1] Schema name=tiny
[junit4] 2> 2398010 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398010 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.CoreContainer Creating SolrCore 'badConfig1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badConfig1, trusted=true
[junit4] 2> 2398010 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.badConfig1' (registry 'solr.core.badConfig1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398010 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398011 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore [[badConfig1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badConfig1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badConfig1/data/]
[junit4] 2> 2398013 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=77.9404296875, floorSegmentMB=1.7822265625, forceMergeDeletesPctAllowed=15.804147445662002, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 2398093 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398093 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398094 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=682579959, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2398094 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.SolrIndexSearcher Opening [Searcher@39715782[badConfig1] main]
[junit4] 2> 2398094 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badConfig1/conf
[junit4] 2> 2398095 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398096 INFO (searcherExecutor-7951-thread-1-processing-x:badConfig1) [ x:badConfig1] o.a.s.c.SolrCore [badConfig1] Registered new searcher Searcher@39715782[badConfig1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398103 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398118 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.IndexSchema [badConfig2] Schema name=tiny
[junit4] 2> 2398120 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398120 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.CoreContainer Creating SolrCore 'badConfig2' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badConfig2, trusted=true
[junit4] 2> 2398120 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.badConfig2' (registry 'solr.core.badConfig2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398121 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398121 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore [[badConfig2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badConfig2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badConfig2/data/]
[junit4] 2> 2398124 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=40, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 2398182 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398182 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398184 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=48, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=79.646484375, floorSegmentMB=1.693359375, forceMergeDeletesPctAllowed=19.322117635995095, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 2398184 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.SolrIndexSearcher Opening [Searcher@46febdbf[badConfig2] main]
[junit4] 2> 2398185 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badConfig2/conf
[junit4] 2> 2398185 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398186 INFO (searcherExecutor-7955-thread-1-processing-x:badConfig1) [ x:badConfig1] o.a.s.c.SolrCore [badConfig2] Registered new searcher Searcher@46febdbf[badConfig2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398196 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398202 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.IndexSchema [badSchema1] Schema name=tiny
[junit4] 2> 2398205 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398205 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.CoreContainer Creating SolrCore 'badSchema1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badSchema1, trusted=true
[junit4] 2> 2398205 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.badSchema1' (registry 'solr.core.badSchema1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398205 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398205 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore [[badSchema1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badSchema1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badSchema1/data/]
[junit4] 2> 2398208 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1216773894, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2398286 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398286 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398287 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=16, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.14602884096195298]
[junit4] 2> 2398288 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2c87904[badSchema1] main]
[junit4] 2> 2398288 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badSchema1/conf
[junit4] 2> 2398289 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398290 INFO (searcherExecutor-7959-thread-1-processing-x:badConfig1) [ x:badConfig1] o.a.s.c.SolrCore [badSchema1] Registered new searcher Searcher@2c87904[badSchema1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398300 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398325 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.IndexSchema [badSchema2] Schema name=tiny
[junit4] 2> 2398327 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398327 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.CoreContainer Creating SolrCore 'badSchema2' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badSchema2, trusted=true
[junit4] 2> 2398328 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.badSchema2' (registry 'solr.core.badSchema2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398328 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398328 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore [[badSchema2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badSchema2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badSchema2/data/]
[junit4] 2> 2398329 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=11, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=63.4462890625, floorSegmentMB=1.6611328125, forceMergeDeletesPctAllowed=20.648536144467826, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 2398371 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398371 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398372 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=2, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=97.765625, floorSegmentMB=1.044921875, forceMergeDeletesPctAllowed=9.45649858313776, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 2398373 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.SolrIndexSearcher Opening [Searcher@296d0e14[badSchema2] main]
[junit4] 2> 2398373 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-001/badSchema2/conf
[junit4] 2> 2398373 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398374 INFO (searcherExecutor-7963-thread-1-processing-x:badConfig1) [ x:badConfig1] o.a.s.c.SolrCore [badSchema2] Registered new searcher Searcher@296d0e14[badSchema2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398375 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 2398375 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@44bbfa95 commitCommandVersion:0
[junit4] 2> 2398393 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.SolrIndexSearcher Opening [Searcher@d06e8dd[core1] main]
[junit4] 2> 2398394 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 2398394 INFO (searcherExecutor-7938-thread-1-processing-x:badConfig1) [ x:badConfig1] o.a.s.c.SolrCore [core1] Registered new searcher Searcher@d06e8dd[core1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.1.0):c7)))}
[junit4] 2> 2398395 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [core1] webapp=null path=null params={q={!prefix+f%3Dv_t}hel&wt=xml} hits=2 status=0 QTime=0
[junit4] 2> 2398396 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [core1] webapp=null path=null params={q={!raw+f%3Dv_t}hello&wt=xml} hits=2 status=0 QTime=0
[junit4] 2> 2398396 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [core1] webapp=null path=null params={q={!raw+f%3Dv_t}Hello&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2398397 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [core1] webapp=null path=null params={q={!raw+f%3Dv_f}1.5&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2398397 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 2398397 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@7d746f24 commitCommandVersion:0
[junit4] 2> 2398403 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.SolrIndexSearcher Opening [Searcher@16b12b[badConfig1] main]
[junit4] 2> 2398403 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 2398404 INFO (searcherExecutor-7951-thread-1-processing-x:badConfig1) [ x:badConfig1] o.a.s.c.SolrCore [badConfig1] Registered new searcher Searcher@16b12b[badConfig1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c7)))}
[junit4] 2> 2398404 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [badConfig1] webapp=null path=null params={q={!prefix+f%3Dv_t}hel&wt=xml} hits=2 status=0 QTime=0
[junit4] 2> 2398406 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [badConfig1] webapp=null path=null params={q={!raw+f%3Dv_t}hello&wt=xml} hits=2 status=0 QTime=0
[junit4] 2> 2398406 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [badConfig1] webapp=null path=null params={q={!raw+f%3Dv_t}Hello&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2398406 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [badConfig1] webapp=null path=null params={q={!raw+f%3Dv_f}1.5&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2398406 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 2398407 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@36f27edd commitCommandVersion:0
[junit4] 2> 2398412 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.SolrIndexSearcher Opening [Searcher@382424fc[badSchema1] main]
[junit4] 2> 2398412 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 2398413 INFO (searcherExecutor-7959-thread-1-processing-x:badConfig1) [ x:badConfig1] o.a.s.c.SolrCore [badSchema1] Registered new searcher Searcher@382424fc[badSchema1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c7)))}
[junit4] 2> 2398414 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [badSchema1] webapp=null path=null params={q={!prefix+f%3Dv_t}hel&wt=xml} hits=2 status=0 QTime=0
[junit4] 2> 2398414 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [badSchema1] webapp=null path=null params={q={!raw+f%3Dv_t}hello&wt=xml} hits=2 status=0 QTime=0
[junit4] 2> 2398414 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [badSchema1] webapp=null path=null params={q={!raw+f%3Dv_t}Hello&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2398414 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [badSchema1] webapp=null path=null params={q={!raw+f%3Dv_f}1.5&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2398414 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.CoreContainer Shutting down CoreContainer instance=876248317
[junit4] 2> 2398414 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 2398414 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1adef5ee: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@12d2aaf4
[junit4] 2> 2398417 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 2398417 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7735f46c: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@6af08c7b
[junit4] 2> 2398419 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 2398420 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@10a021fe: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2d72aae6
[junit4] 2> 2398420 INFO (coreCloseExecutor-7973-thread-1-processing-x:badConfig1) [ x:core2] o.a.s.c.SolrCore [core2] CLOSING SolrCore org.apache.solr.core.SolrCore@3e243fb8
[junit4] 2> 2398420 INFO (coreCloseExecutor-7973-thread-1-processing-x:badConfig1) [ x:core2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.core2, tag=1042563000
[junit4] 2> 2398420 INFO (coreCloseExecutor-7973-thread-1-processing-x:badConfig1) [ x:core2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@75aa5659: rootName = null, domain = solr.core.core2, service url = null, agent id = null] for registry solr.core.core2 / com.codahale.metrics.MetricRegistry@3db1e460
[junit4] 2> 2398420 INFO (coreCloseExecutor-7973-thread-2-processing-x:badConfig1) [ x:core1] o.a.s.c.SolrCore [core1] CLOSING SolrCore org.apache.solr.core.SolrCore@67208411
[junit4] 2> 2398421 INFO (coreCloseExecutor-7973-thread-5-processing-x:badConfig1) [ x:badConfig1] o.a.s.c.SolrCore [badConfig1] CLOSING SolrCore org.apache.solr.core.SolrCore@4e60b10d
[junit4] 2> 2398421 INFO (coreCloseExecutor-7973-thread-3-processing-x:badConfig1) [ x:badSchema1] o.a.s.c.SolrCore [badSchema1] CLOSING SolrCore org.apache.solr.core.SolrCore@224c54fe
[junit4] 2> 2398422 INFO (coreCloseExecutor-7973-thread-6-processing-x:badConfig1) [ x:badConfig2] o.a.s.c.SolrCore [badConfig2] CLOSING SolrCore org.apache.solr.core.SolrCore@41667c95
[junit4] 2> 2398422 INFO (coreCloseExecutor-7973-thread-4-processing-x:badConfig1) [ x:badSchema2] o.a.s.c.SolrCore [badSchema2] CLOSING SolrCore org.apache.solr.core.SolrCore@4296df82
[junit4] 2> 2398425 INFO (coreCloseExecutor-7973-thread-2-processing-x:badConfig1) [ x:core1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.core1, tag=1730184209
[junit4] 2> 2398426 INFO (coreCloseExecutor-7973-thread-2-processing-x:badConfig1) [ x:core1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@739915af: rootName = null, domain = solr.core.core1, service url = null, agent id = null] for registry solr.core.core1 / com.codahale.metrics.MetricRegistry@16381b71
[junit4] 2> 2398431 INFO (coreCloseExecutor-7973-thread-5-processing-x:badConfig1) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.badConfig1, tag=1314959629
[junit4] 2> 2398431 INFO (coreCloseExecutor-7973-thread-5-processing-x:badConfig1) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3fe24030: rootName = null, domain = solr.core.badConfig1, service url = null, agent id = null] for registry solr.core.badConfig1 / com.codahale.metrics.MetricRegistry@377e245
[junit4] 2> 2398437 INFO (coreCloseExecutor-7973-thread-3-processing-x:badConfig1) [ x:badSchema1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.badSchema1, tag=575427838
[junit4] 2> 2398438 INFO (coreCloseExecutor-7973-thread-3-processing-x:badConfig1) [ x:badSchema1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@e903c04: rootName = null, domain = solr.core.badSchema1, service url = null, agent id = null] for registry solr.core.badSchema1 / com.codahale.metrics.MetricRegistry@5de49abe
[junit4] 2> 2398445 INFO (coreCloseExecutor-7973-thread-6-processing-x:badConfig1) [ x:badConfig2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.badConfig2, tag=1097235605
[junit4] 2> 2398446 INFO (coreCloseExecutor-7973-thread-6-processing-x:badConfig1) [ x:badConfig2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2f94ddd3: rootName = null, domain = solr.core.badConfig2, service url = null, agent id = null] for registry solr.core.badConfig2 / com.codahale.metrics.MetricRegistry@148879f7
[junit4] 2> 2398453 INFO (coreCloseExecutor-7973-thread-4-processing-x:badConfig1) [ x:badSchema2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.badSchema2, tag=1117183874
[junit4] 2> 2398453 INFO (coreCloseExecutor-7973-thread-4-processing-x:badConfig1) [ x:badSchema2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@31219675: rootName = null, domain = solr.core.badSchema2, service url = null, agent id = null] for registry solr.core.badSchema2 / com.codahale.metrics.MetricRegistry@3e5c774d
[junit4] 2> 2398460 INFO (TEST-TestLazyCores.testBadConfigsGenerateErrors-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.SolrTestCaseJ4 ###Ending testBadConfigsGenerateErrors
[junit4] 2> 2398463 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.SolrTestCaseJ4 ###Starting testLazySearch
[junit4] 2> 2398475 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/solr.xml
[junit4] 2> 2398480 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2398519 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398524 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398524 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398526 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2398535 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398542 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398542 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398550 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=tiny
[junit4] 2> 2398551 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.s.IndexSchema [collection2] Schema name=tiny
[junit4] 2> 2398551 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.s.IndexSchema [collection5] Schema name=tiny
[junit4] 2> 2398552 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398552 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/./collection1, trusted=true
[junit4] 2> 2398552 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398552 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398552 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/./collection1/data/]
[junit4] 2> 2398553 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398555 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.c.CoreContainer Creating SolrCore 'collection2' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/./collection2, trusted=true
[junit4] 2> 2398555 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection2' (registry 'solr.core.collection2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398555 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398555 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.c.SolrCore [[collection2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/collection2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/./collection2/data/]
[junit4] 2> 2398554 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2398554 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398556 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.c.CoreContainer Creating SolrCore 'collection5' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/./collection5, trusted=true
[junit4] 2> 2398556 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection5' (registry 'solr.core.collection5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398556 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398556 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.c.SolrCore [[collection5] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/collection5], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/./collection5/data/]
[junit4] 2> 2398557 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2398560 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2398635 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398635 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398636 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1649012097, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2398636 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398636 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398637 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1649012097, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2398637 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.s.SolrIndexSearcher Opening [Searcher@77320298[collection2] main]
[junit4] 2> 2398638 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/collection2/conf
[junit4] 2> 2398638 INFO (coreLoadExecutor-7980-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398639 INFO (searcherExecutor-7982-thread-1-processing-x:collection2) [ x:collection2] o.a.s.c.SolrCore [collection2] Registered new searcher Searcher@77320298[collection2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398640 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.s.SolrIndexSearcher Opening [Searcher@191aa8dc[collection5] main]
[junit4] 2> 2398640 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/collection5/conf
[junit4] 2> 2398641 INFO (coreLoadExecutor-7980-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398642 INFO (searcherExecutor-7983-thread-1-processing-x:collection5) [ x:collection5] o.a.s.c.SolrCore [collection5] Registered new searcher Searcher@191aa8dc[collection5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398648 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398648 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398649 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1649012097, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2398650 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@114bc83c[collection1] main]
[junit4] 2> 2398650 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/collection1/conf
[junit4] 2> 2398650 INFO (coreLoadExecutor-7980-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398651 INFO (searcherExecutor-7981-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@114bc83c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398659 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398664 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.IndexSchema [collection4] Schema name=tiny
[junit4] 2> 2398666 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398666 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.CoreContainer Creating SolrCore 'collection4' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/./collection4, trusted=true
[junit4] 2> 2398667 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection4' (registry 'solr.core.collection4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398667 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398667 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore [[collection4] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/collection4], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/./collection4/data/]
[junit4] 2> 2398669 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@5177373e
[junit4] 2> 2398709 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398709 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398710 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=21, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.734778900135656]
[junit4] 2> 2398710 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.SolrIndexSearcher Opening [Searcher@78afded9[collection4] main]
[junit4] 2> 2398710 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-002/collection4/conf
[junit4] 2> 2398711 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398711 INFO (searcherExecutor-7993-thread-1-processing-x:badConfig1) [ x:badConfig1] o.a.s.c.SolrCore [collection4] Registered new searcher Searcher@78afded9[collection4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398712 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 2398712 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@756aa941 commitCommandVersion:0
[junit4] 2> 2398720 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.s.SolrIndexSearcher Opening [Searcher@4b10af2c[collection4] main]
[junit4] 2> 2398720 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 2398721 INFO (searcherExecutor-7993-thread-1-processing-x:badConfig1) [ x:badConfig1] o.a.s.c.SolrCore [collection4] Registered new searcher Searcher@4b10af2c[collection4] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c7)))}
[junit4] 2> 2398722 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [collection4] webapp=null path=null params={q={!prefix+f%3Dv_t}hel&wt=xml} hits=2 status=0 QTime=0
[junit4] 2> 2398722 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [collection4] webapp=null path=null params={q={!raw+f%3Dv_t}hello&wt=xml} hits=2 status=0 QTime=0
[junit4] 2> 2398723 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [collection4] webapp=null path=null params={q={!raw+f%3Dv_t}Hello&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2398723 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [collection4] webapp=null path=null params={q={!raw+f%3Dv_f}1.5&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2398723 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.S.Request [collection1] webapp=null path=null params={q={!raw+f%3Dv_t}hello&wt=xml} hits=0 status=0 QTime=0
[junit4] 2> 2398723 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1800507311
[junit4] 2> 2398723 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 2398723 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@56cbde23: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@409286df
[junit4] 2> 2398726 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 2398726 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@53c51b57: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@6af08c7b
[junit4] 2> 2398729 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 2398729 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@79c8bf9b: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2d72aae6
[junit4] 2> 2398729 INFO (coreCloseExecutor-7997-thread-1-processing-x:badConfig1) [ x:collection5] o.a.s.c.SolrCore [collection5] CLOSING SolrCore org.apache.solr.core.SolrCore@36159608
[junit4] 2> 2398729 INFO (coreCloseExecutor-7997-thread-2-processing-x:badConfig1) [ x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@3a1a9cbd
[junit4] 2> 2398729 INFO (coreCloseExecutor-7997-thread-1-processing-x:badConfig1) [ x:collection5] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection5, tag=907384328
[junit4] 2> 2398729 INFO (coreCloseExecutor-7997-thread-3-processing-x:badConfig1) [ x:collection4] o.a.s.c.SolrCore [collection4] CLOSING SolrCore org.apache.solr.core.SolrCore@100275f4
[junit4] 2> 2398729 INFO (coreCloseExecutor-7997-thread-1-processing-x:badConfig1) [ x:collection5] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@c9269ad: rootName = null, domain = solr.core.collection5, service url = null, agent id = null] for registry solr.core.collection5 / com.codahale.metrics.MetricRegistry@5220889a
[junit4] 2> 2398729 INFO (coreCloseExecutor-7997-thread-4-processing-x:badConfig1) [ x:collection2] o.a.s.c.SolrCore [collection2] CLOSING SolrCore org.apache.solr.core.SolrCore@7cc6ef35
[junit4] 2> 2398735 INFO (coreCloseExecutor-7997-thread-2-processing-x:badConfig1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=974822589
[junit4] 2> 2398735 INFO (coreCloseExecutor-7997-thread-2-processing-x:badConfig1) [ x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@242b504d: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@5c2f6964
[junit4] 2> 2398742 INFO (coreCloseExecutor-7997-thread-3-processing-x:badConfig1) [ x:collection4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection4, tag=268596724
[junit4] 2> 2398743 INFO (coreCloseExecutor-7997-thread-3-processing-x:badConfig1) [ x:collection4] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@24bc55ad: rootName = null, domain = solr.core.collection4, service url = null, agent id = null] for registry solr.core.collection4 / com.codahale.metrics.MetricRegistry@52726260
[junit4] 2> 2398751 INFO (coreCloseExecutor-7997-thread-4-processing-x:badConfig1) [ x:collection2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection2, tag=2093412149
[junit4] 2> 2398752 INFO (coreCloseExecutor-7997-thread-4-processing-x:badConfig1) [ x:collection2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3d0aaae3: rootName = null, domain = solr.core.collection2, service url = null, agent id = null] for registry solr.core.collection2 / com.codahale.metrics.MetricRegistry@4a59c1f7
[junit4] 2> 2398758 INFO (TEST-TestLazyCores.testLazySearch-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.SolrTestCaseJ4 ###Ending testLazySearch
[junit4] 2> 2398760 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.SolrTestCaseJ4 ###Starting testMidUseUnload
[junit4] 2> 2398779 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/solr.xml
[junit4] 2> 2398784 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2398814 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398819 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398820 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398821 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2398837 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398837 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398838 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2398844 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.s.IndexSchema [collection5] Schema name=tiny
[junit4] 2> 2398845 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.s.IndexSchema [collection2] Schema name=tiny
[junit4] 2> 2398846 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=tiny
[junit4] 2> 2398846 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398846 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.c.CoreContainer Creating SolrCore 'collection5' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/./collection5, trusted=true
[junit4] 2> 2398847 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398847 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.c.CoreContainer Creating SolrCore 'collection2' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/./collection2, trusted=true
[junit4] 2> 2398847 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection5' (registry 'solr.core.collection5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398847 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection2' (registry 'solr.core.collection2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398847 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398847 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398847 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.c.SolrCore [[collection5] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/collection5], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/./collection5/data/]
[junit4] 2> 2398847 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.c.SolrCore [[collection2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/collection2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/./collection2/data/]
[junit4] 2> 2398849 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
[junit4] 2> 2398849 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2398849 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2398849 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/./collection1, trusted=true
[junit4] 2> 2398850 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1' (registry 'solr.core.collection1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@46fd6cf6
[junit4] 2> 2398850 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2398851 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/./collection1/data/]
[junit4] 2> 2398853 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 2398920 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398920 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398921 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1649012097, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2398921 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.s.SolrIndexSearcher Opening [Searcher@6fd1c3ae[collection2] main]
[junit4] 2> 2398921 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/collection2/conf
[junit4] 2> 2398922 INFO (coreLoadExecutor-8004-thread-2-processing-x:badConfig1) [ x:collection2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398922 INFO (searcherExecutor-8006-thread-1-processing-x:collection2) [ x:collection2] o.a.s.c.SolrCore [collection2] Registered new searcher Searcher@6fd1c3ae[collection2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398928 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398928 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398929 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1649012097, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2398929 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.s.SolrIndexSearcher Opening [Searcher@8a679a8[collection5] main]
[junit4] 2> 2398930 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/collection5/conf
[junit4] 2> 2398930 INFO (coreLoadExecutor-8004-thread-3-processing-x:badConfig1) [ x:collection5] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398931 INFO (searcherExecutor-8005-thread-1-processing-x:collection5) [ x:collection5] o.a.s.c.SolrCore [collection5] Registered new searcher Searcher@8a679a8[collection5] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398932 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2398932 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2398933 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1649012097, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
[junit4] 2> 2398934 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@31f0e418[collection1] main]
[junit4] 2> 2398934 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestLazyCores_29196813FF046BA8-001/tempDir-003/collection1/conf
[junit4] 2> 2398935 INFO (coreLoadExecutor-8004-thread-1-processing-x:badConfig1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2398936 INFO (searcherExecutor-8007-thread-1-processing-x:collection1) [ x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@31f0e418[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2398938 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=null
[junit4] 2> 2398939 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2ab83416: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@3acdafd6
[junit4] 2> 2398946 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2399046 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2399147 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2399247 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2399347 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2399447 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2399548 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2399648 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2399748 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2399848 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2399948 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2400048 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2400148 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2400248 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2400349 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2400449 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2400549 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2400649 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2400750 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2400851 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2400952 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2401052 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2401152 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2401252 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2401352 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2401452 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2401553 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2401653 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2401753 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2401853 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2401953 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2402054 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2402154 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2402254 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2402354 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2402454 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 100 ms before checking again.
[junit4] 2> 2402554 INFO (TEST-TestLazyCores.testMidUseUnload-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.c.SolrCore Core collection1 is not yet closed, waiting 1
[...truncated too long message...]
/ com.codahale.metrics.MetricRegistry@363b6d94
[junit4] 2> 9618443 INFO (coreCloseExecutor-8183-thread-3-processing-x:badConfig1) [ x:collection4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection4, tag=69331815
[junit4] 2> 9618443 INFO (coreCloseExecutor-8183-thread-3-processing-x:badConfig1) [ x:collection4] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@12667e2e: rootName = null, domain = solr.core.collection4, service url = null, agent id = null] for registry solr.core.collection4 / com.codahale.metrics.MetricRegistry@2d81c0cb
[junit4] 2> 9618448 INFO (coreCloseExecutor-8183-thread-5-processing-x:badConfig1) [ x:collection7] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection7, tag=1795001538
[junit4] 2> 9618448 INFO (coreCloseExecutor-8183-thread-5-processing-x:badConfig1) [ x:collection7] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7658c14c: rootName = null, domain = solr.core.collection7, service url = null, agent id = null] for registry solr.core.collection7 / com.codahale.metrics.MetricRegistry@5626524d
[junit4] 2> 9618453 INFO (coreCloseExecutor-8183-thread-7-processing-x:badConfig1) [ x:collection9] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection9, tag=1044871522
[junit4] 2> 9618453 INFO (coreCloseExecutor-8183-thread-7-processing-x:badConfig1) [ x:collection9] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@33440012: rootName = null, domain = solr.core.collection9, service url = null, agent id = null] for registry solr.core.collection9 / com.codahale.metrics.MetricRegistry@78301706
[junit4] 2> 9618461 INFO (TEST-TestLazyCores.testCachingLimit-seed#[29196813FF046BA8]) [ x:badConfig1] o.a.s.SolrTestCaseJ4 ###Ending testCachingLimit
[junit4] 2> 9618462 INFO (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ x:badConfig1] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> 9618462 INFO (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ x:badConfig1] o.a.s.c.CoreContainer Shutting down CoreContainer instance=568506576
[junit4] 2> 9618462 INFO (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 9618462 INFO (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 9618462 INFO (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ x:badConfig1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 9618462 INFO (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ x:badConfig1] o.a.s.SolrTestCaseJ4 Waiting for all tracked resources to be released
[junit4] 2> 9618502 INFO (SUITE-TestLazyCores-seed#[29196813FF046BA8]-worker) [ x:badConfig1] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Sep 12, 2017 10:44:31 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=20936, name=searcherExecutor-8119-thread-1, state=WAITING, group=TGRP-TestLazyCores]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=en-SG, timezone=Asia/Tokyo
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=2,free=225441728,total=530579456
[junit4] 2> NOTE: All tests run in this JVM: [DirectSolrConnectionTest, TestRequestStatusCollectionAPI, CurrencyFieldTypeTest, DocValuesTest, TestSchemaNameResource, CoreMergeIndexesAdminHandlerTest, TestHdfsUpdateLog, TestPHPSerializedResponseWriter, TestMaxTokenLenTokenizer, TestDynamicFieldCollectionResource, TestHalfAndHalfDocValues, SpellCheckCollatorWithCollapseTest, LeaderElectionContextKeyTest, TestCoreContainer, TestSizeLimitedDistributedMap, MinimalSchemaTest, TestPrepRecovery, TestXmlQParserPlugin, HdfsThreadLeakTest, TestLegacyFieldReuse, TestSolrCoreProperties, DirectUpdateHandlerOptimizeTest, DistribCursorPagingTest, TestTrieFacet, DisMaxRequestHandlerTest, TestOmitPositions, TestFieldCache, ConfigSetsAPITest, SimpleFacetsTest, TestCloudJSONFacetJoinDomain, BufferStoreTest, TestInitParams, SolrIndexSplitterTest, TestRawResponseWriter, TestImpersonationWithHadoopAuth, JvmMetricsTest, SpatialHeatmapFacetsTest, TestSha256AuthenticationProvider, CdcrReplicationHandlerTest, TestManagedSynonymGraphFilterFactory, CheckHdfsIndexTest, ResourceLoaderTest, TestRestoreCore, TestDistributedSearch, SolrShardReporterTest, TestStandardQParsers, TestReloadAndDeleteDocs, CollectionTooManyReplicasTest, TestDeleteCollectionOnDownNodes, TestLeaderElectionZkExpiry, TestRTimerTree, HdfsRestartWhileUpdatingTest, TestSimpleTextCodec, CloudMLTQParserTest, AssignBackwardCompatibilityTest, SolrCoreTest, DistributedSpellCheckComponentTest, SuggesterWFSTTest, SparseHLLTest, UUIDFieldTest, DocExpirationUpdateProcessorFactoryTest, ConfigureRecoveryStrategyTest, TestGeoJSONResponseWriter, TestSolrCoreParser, TestManagedStopFilterFactory, TestJmxIntegration, TestEmbeddedSolrServerConstructors, NumericFieldsTest, TestFreeTextSuggestions, TestExpandComponent, SharedFSAutoReplicaFailoverUtilsTest, TestZkAclsWithHadoopAuth, TestReversedWildcardFilterFactory, TestPhraseSuggestions, TestJsonFacets, TestNonDefinedSimilarityFactory, TestStressCloudBlindAtomicUpdates, TestSmileRequest, BooleanFieldTest, TestConfigSetImmutable, TestRandomDVFaceting, SolrInfoBeanTest, DateMathParserTest, OverseerTest, TestHashPartitioner, TestFaceting, DateFieldTest, CdcrBootstrapTest, ChaosMonkeySafeLeaderWithPullReplicasTest, TestStressReorder, TestRandomRequestDistribution, TestConfigSetsAPIZkFailure, TestIBSimilarityFactory, LargeFieldTest, TestFieldCacheSort, TestLFUCache, CreateCollectionCleanupTest, OutOfBoxZkACLAndCredentialsProvidersTest, HighlighterConfigTest, TestHashQParserPlugin, TestSolrQueryResponse, ClassificationUpdateProcessorFactoryTest, HardAutoCommitTest, DeleteShardTest, SimplePostToolTest, ClassificationUpdateProcessorTest, DistribJoinFromCollectionTest, MoveReplicaHDFSUlogDirTest, AtomicUpdateProcessorFactoryTest, TestAtomicUpdateErrorCases, TestGroupingSearch, VersionInfoTest, TestPseudoReturnFields, TestManagedResourceStorage, SuggestComponentTest, HdfsBasicDistributedZk2Test, TestCollectionAPI, TestConfigReload, CleanupOldIndexTest, SolrJmxReporterCloudTest, TestSolrCoreSnapshots, AlternateDirectoryTest, TestStressVersions, SoftAutoCommitTest, LeaderFailureAfterFreshStartTest, TestCloudDeleteByQuery, TestImplicitCoreProperties, DistributedFacetPivotLargeTest, TestScoreJoinQPScore, SolrTestCaseJ4Test, TestFuzzyAnalyzedSuggestions, AnalyticsQueryTest, BlockJoinFacetRandomTest, ForceLeaderTest, SyncSliceTest, TestNestedDocsSort, TestCodecSupport, TestUseDocValuesAsStored2, TestCoreAdminApis, BJQParserTest, ZkStateReaderTest, SuggesterTest, V2ApiIntegrationTest, ResponseLogComponentTest, CoreAdminCreateDiscoverTest, XsltUpdateRequestHandlerTest, TestBulkSchemaAPI, PrimitiveFieldTypeTest, ReplaceNodeTest, TestDistribIDF, TestManagedSchemaAPI, MissingSegmentRecoveryTest, TestReplicationHandlerBackup, TestSchemaSimilarityResource, TestStressRecovery, TestRecoveryHdfs, TestCursorMarkWithoutUniqueKey, TestSerializedLuceneMatchVersion, TestTolerantUpdateProcessorCloud, TestCustomStream, TestCustomDocTransformer, TestDownShardTolerantSearch, TestBackupRepositoryFactory, ReplicaListTransformerTest, SolrMetricReporterTest, QueryElevationComponentTest, TestQuerySenderNoQuery, TestMacros, UUIDUpdateProcessorFallbackTest, HLLSerializationTest, HdfsRecoverLeaseTest, TestBlendedInfixSuggestions, DistributedVersionInfoTest, ResponseHeaderTest, DocValuesMissingTest, OutputWriterTest, UnloadDistributedZkTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, ClusterStateUpdateTest, LeaderElectionTest, ZkSolrClientTest, ZkCLITest, TestDistributedGrouping, TestRecovery, ZkControllerTest, TestRealTimeGet, DistributedTermsComponentTest, TestRangeQuery, TestSolr4Spatial, SolrCmdDistributorTest, PeerSyncTest, BadIndexSchemaTest, TestSort, TestFiltering, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestLazyCores -Dtests.seed=29196813FF046BA8 -Dtests.slow=true -Dtests.locale=en-SG -Dtests.timezone=Asia/Tokyo -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J0 | TestLazyCores (suite) <<<
[junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4] > at __randomizedtesting.SeedInfo.seed([29196813FF046BA8]:0)
[junit4] Completed [731/731 (1!)] on J0 in 7223.25s, 7 tests, 2 errors <<< FAILURES!
[...truncated 46256 lines...]
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 173 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/173/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.TestPullReplica.testKillLeader
Error Message:
Replica state not updated in cluster state null Live Nodes: [127.0.0.1:40037_solr, 127.0.0.1:57109_solr] Last available state: DocCollection(pull_replica_test_kill_leader//collections/pull_replica_test_kill_leader/state.json/6)={ "pullReplicas":"1", "replicationFactor":"1", "shards":{"shard1":{ "range":"80000000-7fffffff", "state":"active", "replicas":{ "core_node3":{ "core":"pull_replica_test_kill_leader_shard1_replica_n1", "base_url":"http://127.0.0.1:57109/solr", "node_name":"127.0.0.1:57109_solr", "state":"down", "type":"NRT", "leader":"true"}, "core_node4":{ "core":"pull_replica_test_kill_leader_shard1_replica_p2", "base_url":"http://127.0.0.1:40037/solr", "node_name":"127.0.0.1:40037_solr", "state":"active", "type":"PULL"}}}}, "router":{"name":"compositeId"}, "maxShardsPerNode":"100", "autoAddReplicas":"false", "nrtReplicas":"1", "tlogReplicas":"0"}
Stack Trace:
java.lang.AssertionError: Replica state not updated in cluster state
null
Live Nodes: [127.0.0.1:40037_solr, 127.0.0.1:57109_solr]
Last available state: DocCollection(pull_replica_test_kill_leader//collections/pull_replica_test_kill_leader/state.json/6)={
"pullReplicas":"1",
"replicationFactor":"1",
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{
"core_node3":{
"core":"pull_replica_test_kill_leader_shard1_replica_n1",
"base_url":"http://127.0.0.1:57109/solr",
"node_name":"127.0.0.1:57109_solr",
"state":"down",
"type":"NRT",
"leader":"true"},
"core_node4":{
"core":"pull_replica_test_kill_leader_shard1_replica_p2",
"base_url":"http://127.0.0.1:40037/solr",
"node_name":"127.0.0.1:40037_solr",
"state":"active",
"type":"PULL"}}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"100",
"autoAddReplicas":"false",
"nrtReplicas":"1",
"tlogReplicas":"0"}
at __randomizedtesting.SeedInfo.seed([BA5BFFA382390CBC:F34D0B17E08298EA]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:269)
at org.apache.solr.cloud.TestPullReplica.doTestNoLeader(TestPullReplica.java:401)
at org.apache.solr.cloud.TestPullReplica.testKillLeader(TestPullReplica.java:290)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12359 lines...]
[junit4] Suite: org.apache.solr.cloud.TestPullReplica
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/init-core-data-001
[junit4] 2> 1529468 WARN (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=15 numCloses=15
[junit4] 2> 1529468 INFO (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1529469 INFO (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 1529469 INFO (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1529470 INFO (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001
[junit4] 2> 1529470 INFO (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1529470 INFO (Thread-3929) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1529470 INFO (Thread-3929) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1529478 ERROR (Thread-3929) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1529570 INFO (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:41972
[junit4] 2> 1529612 INFO (jetty-launcher-2563-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1529615 INFO (jetty-launcher-2563-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1529618 INFO (jetty-launcher-2563-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@74283d5b{/solr,null,AVAILABLE}
[junit4] 2> 1529622 INFO (jetty-launcher-2563-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f769c87{/solr,null,AVAILABLE}
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4811ada7{HTTP/1.1,[http/1.1]}{127.0.0.1:40037}
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-1) [ ] o.e.j.s.Server Started @1532375ms
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@537e0269{HTTP/1.1,[http/1.1]}{127.0.0.1:57109}
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40037}
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-2) [ ] o.e.j.s.Server Started @1532375ms
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57109}
[junit4] 2> 1529623 ERROR (jetty-launcher-2563-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1529623 ERROR (jetty-launcher-2563-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1529623 INFO (jetty-launcher-2563-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 1529627 INFO (jetty-launcher-2563-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 1529628 INFO (jetty-launcher-2563-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-12T13:11:17.863Z
[junit4] 2> 1529627 INFO (jetty-launcher-2563-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-12T13:11:17.862Z
[junit4] 2> 1529652 INFO (jetty-launcher-2563-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1529653 INFO (jetty-launcher-2563-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1529659 INFO (jetty-launcher-2563-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41972/solr
[junit4] 2> 1529661 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e76371fcb0001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1529684 INFO (jetty-launcher-2563-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41972/solr
[junit4] 2> 1529728 INFO (jetty-launcher-2563-thread-2) [n:127.0.0.1:57109_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1529730 INFO (jetty-launcher-2563-thread-2) [n:127.0.0.1:57109_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:57109_solr
[junit4] 2> 1529733 INFO (jetty-launcher-2563-thread-2) [n:127.0.0.1:57109_solr ] o.a.s.c.Overseer Overseer (id=98646220977405958-127.0.0.1:57109_solr-n_0000000000) starting
[junit4] 2> 1529749 INFO (jetty-launcher-2563-thread-2) [n:127.0.0.1:57109_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57109_solr
[junit4] 2> 1529771 INFO (jetty-launcher-2563-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1529774 INFO (jetty-launcher-2563-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1529776 INFO (jetty-launcher-2563-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40037_solr
[junit4] 2> 1529779 INFO (zkCallback-2576-thread-1-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1529786 INFO (zkCallback-2576-thread-3-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1529790 INFO (zkCallback-2571-thread-1-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1529957 INFO (jetty-launcher-2563-thread-2) [n:127.0.0.1:57109_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57109.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3070e215
[junit4] 2> 1529971 INFO (jetty-launcher-2563-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40037.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3070e215
[junit4] 2> 1529978 INFO (jetty-launcher-2563-thread-2) [n:127.0.0.1:57109_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57109.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3070e215
[junit4] 2> 1529979 INFO (jetty-launcher-2563-thread-2) [n:127.0.0.1:57109_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57109.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3070e215
[junit4] 2> 1529979 INFO (jetty-launcher-2563-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40037.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3070e215
[junit4] 2> 1529979 INFO (jetty-launcher-2563-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40037.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3070e215
[junit4] 2> 1529979 INFO (jetty-launcher-2563-thread-2) [n:127.0.0.1:57109_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001/node2/.
[junit4] 2> 1529980 INFO (jetty-launcher-2563-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001/node1/.
[junit4] 2> 1530073 INFO (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1530074 INFO (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:41972/solr ready
[junit4] 2> 1530080 INFO (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.c.TestPullReplica Using legacyCloud?: false
[junit4] 2> 1530085 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1530087 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 1530264 INFO (TEST-TestPullReplica.testSearchWhileReplicationHappens-seed#[BA5BFFA382390CBC]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testSearchWhileReplicationHappens
[junit4] 2> 1530265 INFO (TEST-TestPullReplica.testSearchWhileReplicationHappens-seed#[BA5BFFA382390CBC]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testSearchWhileReplicationHappens
[junit4] 2> 1530419 INFO (TEST-TestPullReplica.testRealTimeGet-seed#[BA5BFFA382390CBC]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testRealTimeGet
[junit4] 2> 1530420 INFO (qtp12181658-15434) [n:127.0.0.1:40037_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=2&replicationFactor=2&collection.configName=conf&maxShardsPerNode=100&name=pull_replica_test_real_time_get&nrtReplicas=2&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1530427 INFO (OverseerThreadFactory-6136-thread-1-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.CreateCollectionCmd Create collection pull_replica_test_real_time_get
[junit4] 2> 1530428 WARN (OverseerThreadFactory-6136-thread-1-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.CreateCollectionCmd Specified number of replicas of 4 on collection pull_replica_test_real_time_get is higher than the number of Solr instances currently live or live and part of your createNodeSet(2). It's unusual to run two replica of the same slice on the same Solr-instance.
[junit4] 2> 1530543 INFO (OverseerStateUpdate-98646220977405958-127.0.0.1:57109_solr-n_0000000000) [n:127.0.0.1:57109_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"pull_replica_test_real_time_get",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"pull_replica_test_real_time_get_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:40037/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 1530552 INFO (OverseerStateUpdate-98646220977405958-127.0.0.1:57109_solr-n_0000000000) [n:127.0.0.1:57109_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"pull_replica_test_real_time_get",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"pull_replica_test_real_time_get_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:57109/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 1530557 INFO (OverseerStateUpdate-98646220977405958-127.0.0.1:57109_solr-n_0000000000) [n:127.0.0.1:57109_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"pull_replica_test_real_time_get",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"pull_replica_test_real_time_get_shard1_replica_p4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:40037/solr",
[junit4] 2> "type":"PULL"}
[junit4] 2> 1530560 INFO (OverseerStateUpdate-98646220977405958-127.0.0.1:57109_solr-n_0000000000) [n:127.0.0.1:57109_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"pull_replica_test_real_time_get",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"pull_replica_test_real_time_get_shard1_replica_p6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:57109/solr",
[junit4] 2> "type":"PULL"}
[junit4] 2> 1530767 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=pull_replica_test_real_time_get_shard1_replica_n1&action=CREATE&numShards=1&collection=pull_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1530767 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=pull_replica_test_real_time_get_shard1_replica_p4&action=CREATE&numShards=1&collection=pull_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=PULL
[junit4] 2> 1530767 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1530768 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=pull_replica_test_real_time_get_shard1_replica_p6&action=CREATE&numShards=1&collection=pull_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=PULL
[junit4] 2> 1530768 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1530769 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=pull_replica_test_real_time_get_shard1_replica_n2&action=CREATE&numShards=1&collection=pull_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1530880 INFO (zkCallback-2576-thread-2-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1530880 INFO (zkCallback-2576-thread-3-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1530880 INFO (zkCallback-2571-thread-2-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1530881 INFO (zkCallback-2571-thread-1-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1531789 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 1531789 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 1531791 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 1531802 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 1531803 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.s.IndexSchema [pull_replica_test_real_time_get_shard1_replica_p4] Schema name=minimal
[junit4] 2> 1531803 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.s.IndexSchema [pull_replica_test_real_time_get_shard1_replica_n2] Schema name=minimal
[junit4] 2> 1531806 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1531806 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_real_time_get_shard1_replica_p4' using configuration from collection pull_replica_test_real_time_get, trusted=true
[junit4] 2> 1531806 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1531807 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40037.solr.core.pull_replica_test_real_time_get.shard1.replica_p4' (registry 'solr.core.pull_replica_test_real_time_get.shard1.replica_p4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3070e215
[junit4] 2> 1531807 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1531807 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.SolrCore [[pull_replica_test_real_time_get_shard1_replica_p4] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001/node1/pull_replica_test_real_time_get_shard1_replica_p4], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001/node1/./pull_replica_test_real_time_get_shard1_replica_p4/data/]
[junit4] 2> 1531808 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_real_time_get_shard1_replica_n2' using configuration from collection pull_replica_test_real_time_get, trusted=true
[junit4] 2> 1531809 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57109.solr.core.pull_replica_test_real_time_get.shard1.replica_n2' (registry 'solr.core.pull_replica_test_real_time_get.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3070e215
[junit4] 2> 1531809 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1531809 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.SolrCore [[pull_replica_test_real_time_get_shard1_replica_n2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001/node2/pull_replica_test_real_time_get_shard1_replica_n2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001/node2/./pull_replica_test_real_time_get_shard1_replica_n2/data/]
[junit4] 2> 1531809 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.s.IndexSchema [pull_replica_test_real_time_get_shard1_replica_p6] Schema name=minimal
[junit4] 2> 1531811 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1531811 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_real_time_get_shard1_replica_p6' using configuration from collection pull_replica_test_real_time_get, trusted=true
[junit4] 2> 1531812 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57109.solr.core.pull_replica_test_real_time_get.shard1.replica_p6' (registry 'solr.core.pull_replica_test_real_time_get.shard1.replica_p6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3070e215
[junit4] 2> 1531812 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1531812 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.SolrCore [[pull_replica_test_real_time_get_shard1_replica_p6] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001/node2/pull_replica_test_real_time_get_shard1_replica_p6], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001/node2/./pull_replica_test_real_time_get_shard1_replica_p6/data/]
[junit4] 2> 1531819 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.IndexSchema [pull_replica_test_real_time_get_shard1_replica_n1] Schema name=minimal
[junit4] 2> 1531821 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1531821 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'pull_replica_test_real_time_get_shard1_replica_n1' using configuration from collection pull_replica_test_real_time_get, trusted=true
[junit4] 2> 1531821 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40037.solr.core.pull_replica_test_real_time_get.shard1.replica_n1' (registry 'solr.core.pull_replica_test_real_time_get.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3070e215
[junit4] 2> 1531822 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1531822 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrCore [[pull_replica_test_real_time_get_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001/node1/pull_replica_test_real_time_get_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001/tempDir-001/node1/./pull_replica_test_real_time_get_shard1_replica_n1/data/]
[junit4] 2> 1531889 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1531889 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1531890 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1531890 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1531891 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1531891 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1531891 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.s.SolrIndexSearcher Opening [Searcher@38fc80fe[pull_replica_test_real_time_get_shard1_replica_p6] main]
[junit4] 2> 1531892 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@1aee8916[pull_replica_test_real_time_get_shard1_replica_n2] main]
[junit4] 2> 1531893 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1531893 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1531894 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1531895 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1531895 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1531895 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1531895 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1531895 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578339538179194880
[junit4] 2> 1531900 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1531900 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1531901 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1531901 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@c586193[pull_replica_test_real_time_get_shard1_replica_n1] main]
[junit4] 2> 1531902 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.ZkController pull_replica_test_real_time_get_shard1_replica_p6 starting background replication from leader
[junit4] 2> 1531902 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 1531902 INFO (searcherExecutor-6142-thread-1-processing-n:127.0.0.1:57109_solr x:pull_replica_test_real_time_get_shard1_replica_n2 s:shard1 c:pull_replica_test_real_time_get r:core_node5) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.SolrCore [pull_replica_test_real_time_get_shard1_replica_n2] Registered new searcher Searcher@1aee8916[pull_replica_test_real_time_get_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1531902 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1531903 INFO (searcherExecutor-6143-thread-1-processing-n:127.0.0.1:57109_solr x:pull_replica_test_real_time_get_shard1_replica_p6 s:shard1 c:pull_replica_test_real_time_get r:core_node8) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.SolrCore [pull_replica_test_real_time_get_shard1_replica_p6] Registered new searcher Searcher@38fc80fe[pull_replica_test_real_time_get_shard1_replica_p6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1531903 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1531904 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1531905 INFO (searcherExecutor-6144-thread-1-processing-n:127.0.0.1:40037_solr x:pull_replica_test_real_time_get_shard1_replica_n1 s:shard1 c:pull_replica_test_real_time_get r:core_node3) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrCore [pull_replica_test_real_time_get_shard1_replica_n1] Registered new searcher Searcher@c586193[pull_replica_test_real_time_get_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1531905 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578339538189680640
[junit4] 2> 1531906 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 1531908 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 1531912 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1531912 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1531913 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.s.SolrIndexSearcher Opening [Searcher@20f79d8d[pull_replica_test_real_time_get_shard1_replica_p4] main]
[junit4] 2> 1531914 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1531915 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1531915 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 1531915 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.ZkController pull_replica_test_real_time_get_shard1_replica_p4 starting background replication from leader
[junit4] 2> 1531915 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 1531916 INFO (searcherExecutor-6141-thread-1-processing-n:127.0.0.1:40037_solr x:pull_replica_test_real_time_get_shard1_replica_p4 s:shard1 c:pull_replica_test_real_time_get r:core_node7) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.SolrCore [pull_replica_test_real_time_get_shard1_replica_p4] Registered new searcher Searcher@20f79d8d[pull_replica_test_real_time_get_shard1_replica_p4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1531916 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 1532012 INFO (zkCallback-2571-thread-1-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1532012 INFO (zkCallback-2576-thread-3-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1532012 INFO (zkCallback-2576-thread-2-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1532012 INFO (zkCallback-2571-thread-2-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1532411 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1532411 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1532411 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/
[junit4] 2> 1532411 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=pull_replica_test_real_time_get_shard1_replica_n2 url=http://127.0.0.1:57109/solr START replicas=[http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/] nUpdates=100
[junit4] 2> 1532413 INFO (qtp12181658-15431) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1532714 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=pull_replica_test_real_time_get_shard1_replica_n2 url=http://127.0.0.1:57109/solr DONE. We have no versions. sync failed.
[junit4] 2> 1532714 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 1532714 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 1532715 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1532727 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/ shard1
[junit4] 2> 1532834 INFO (zkCallback-2571-thread-2-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1532834 INFO (zkCallback-2576-thread-2-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1532834 INFO (zkCallback-2576-thread-3-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1532834 INFO (zkCallback-2571-thread-1-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1532886 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1532889 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=pull_replica_test_real_time_get_shard1_replica_n2&action=CREATE&numShards=1&collection=pull_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2121
[junit4] 2> 1532910 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=pull_replica_test_real_time_get_shard1_replica_p6&action=CREATE&numShards=1&collection=pull_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=PULL} status=0 QTime=2142
[junit4] 2> 1532913 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=pull_replica_test_real_time_get_shard1_replica_n1&action=CREATE&numShards=1&collection=pull_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2146
[junit4] 2> 1532926 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=pull_replica_test_real_time_get_shard1_replica_p4&action=CREATE&numShards=1&collection=pull_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=PULL} status=0 QTime=2160
[junit4] 2> 1532934 INFO (qtp12181658-15434) [n:127.0.0.1:40037_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1533038 INFO (zkCallback-2576-thread-2-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1533038 INFO (zkCallback-2571-thread-1-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1533038 INFO (zkCallback-2576-thread-3-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1533038 INFO (zkCallback-2571-thread-2-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/pull_replica_test_real_time_get/state.json] for collection [pull_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1533464 INFO (indexFetcher-6162-thread-1) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.IndexFetcher Updated masterUrl to http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/
[junit4] 2> 1533466 INFO (qtp441150007-15445) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1533467 INFO (indexFetcher-6162-thread-1) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1533467 INFO (indexFetcher-6162-thread-1) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1533467 INFO (indexFetcher-6162-thread-1) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1533467 INFO (indexFetcher-6162-thread-1) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1533935 INFO (qtp12181658-15434) [n:127.0.0.1:40037_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=2&replicationFactor=2&collection.configName=conf&maxShardsPerNode=100&name=pull_replica_test_real_time_get&nrtReplicas=2&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2} status=0 QTime=3516
[junit4] 2> 1533961 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/&wt=javabin&version=2}{add=[0 (1578339540340310016)]} 0 1
[junit4] 2> 1533961 INFO (qtp441150007-15447) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2}{add=[0 (1578339540340310016)]} 0 6
[junit4] 2> 1533962 INFO (qtp12181658-15436) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 9
[junit4] 2> 1533963 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&_stateVer_=pull_replica_test_real_time_get:5&ids=0&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533965 INFO (qtp12181658-15431) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533966 INFO (qtp441150007-15438) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533968 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/
[junit4] 2> 1533970 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505221882202&ids=0&isShard=true&shard.url=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533970 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p4] webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=3
[junit4] 2> 1533971 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/
[junit4] 2> 1533972 INFO (qtp441150007-15446) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505221882206&ids=0&isShard=true&shard.url=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533973 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p6] webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 1533974 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/&wt=javabin&version=2}{add=[1 (1578339540359184384)]} 0 0
[junit4] 2> 1533975 INFO (qtp441150007-15445) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[1 (1578339540359184384)]} 0 1
[junit4] 2> 1533975 INFO (qtp441150007-15447) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/|http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/
[junit4] 2> 1533976 INFO (qtp12181658-15434) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&_stateVer_=pull_replica_test_real_time_get:5&omitHeader=true&shards.purpose=1&NOW=1505221882210&ids=1&isShard=true&shard.url=http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/|http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533976 INFO (qtp441150007-15447) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p6] webapp=/solr path=/get params={qt=/get&_stateVer_=pull_replica_test_real_time_get:5&ids=1&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 1533977 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533977 INFO (qtp441150007-15438) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533978 INFO (qtp12181658-15436) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/|http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/
[junit4] 2> 1533979 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505221882213&ids=1&isShard=true&shard.url=http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/|http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533979 INFO (qtp12181658-15436) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p4] webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 1533980 INFO (qtp441150007-15443) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/
[junit4] 2> 1533981 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505221882215&ids=1&isShard=true&shard.url=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533981 INFO (qtp441150007-15443) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p6] webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 1533983 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/&wt=javabin&version=2}{add=[2 (1578339540367572992)]} 0 0
[junit4] 2> 1533983 INFO (qtp441150007-15446) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_p4/&wt=javabin&version=2}{add=[2 (1578339540367572992)]} 0 0
[junit4] 2> 1533983 INFO (qtp12181658-15431) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_p4] webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 1
[junit4] 2> 1533984 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&_stateVer_=pull_replica_test_real_time_get:5&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533984 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533985 INFO (qtp441150007-15445) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533985 INFO (qtp12181658-15434) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/
[junit4] 2> 1533986 INFO (qtp441150007-15447) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505221882220&ids=2&isShard=true&shard.url=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533986 INFO (qtp12181658-15434) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p4] webapp=/solr path=/get params={qt=/get&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533986 INFO (qtp441150007-15438) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/
[junit4] 2> 1533987 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505221882221&ids=2&isShard=true&shard.url=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533987 INFO (qtp441150007-15438) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p6] webapp=/solr path=/get params={qt=/get&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533990 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/&wt=javabin&version=2}{add=[3 (1578339540374913024)]} 0 0
[junit4] 2> 1533990 INFO (qtp441150007-15446) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_p6/&wt=javabin&version=2}{add=[3 (1578339540374913024)]} 0 1
[junit4] 2> 1533990 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.u.p.LogUpdateProcessorFactory [pull_replica_test_real_time_get_shard1_replica_p6] webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 2
[junit4] 2> 1533991 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&_stateVer_=pull_replica_test_real_time_get:5&ids=3&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533991 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=3&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533992 INFO (qtp441150007-15446) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&ids=3&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533992 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/
[junit4] 2> 1533993 INFO (qtp441150007-15445) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505221882227&ids=3&isShard=true&shard.url=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533993 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p4] webapp=/solr path=/get params={qt=/get&ids=3&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 1533994 INFO (qtp441150007-15447) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/|http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/
[junit4] 2> 1533994 INFO (qtp12181658-15436) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505221882229&ids=3&isShard=true&shard.url=http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/|http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533994 INFO (qtp441150007-15447) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p6] webapp=/solr path=/get params={qt=/get&ids=3&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533995 INFO (qtp12181658-15439) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=0&ids=1&ids=2&ids=3&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533996 INFO (qtp441150007-15438) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&ids=0&ids=1&ids=2&ids=3&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533996 INFO (qtp12181658-15431) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/|http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/
[junit4] 2> 1533997 INFO (qtp12181658-15435) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node3 x:pull_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505221882231&ids=0,1,2,3&isShard=true&shard.url=http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/|http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533997 INFO (qtp12181658-15431) [n:127.0.0.1:40037_solr c:pull_replica_test_real_time_get s:shard1 r:core_node7 x:pull_replica_test_real_time_get_shard1_replica_p4] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p4] webapp=/solr path=/get params={qt=/get&ids=0&ids=1&ids=2&ids=3&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533997 INFO (qtp441150007-15443) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/
[junit4] 2> 1533998 INFO (qtp441150007-15444) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node5 x:pull_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1505221882232&ids=0,1,2,3&isShard=true&shard.url=http://127.0.0.1:57109/solr/pull_replica_test_real_time_get_shard1_replica_n2/|http://127.0.0.1:40037/solr/pull_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 1533998 INFO (qtp441150007-15443) [n:127.0.0.1:57109_solr c:pull_replica_test_real_time_get s:shard1 r:core_node8 x:pull_replica_test_real_time_get_shard1_replica_p6] o.a.s.c.S.Request [pull_replica_test_real_time_get_shard1_replica_p6] webapp=/solr path=/get params={qt=/get&ids=0&ids=1&ids=2&ids=3&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1533998 INFO (TEST-TestPullReplica.testRealTimeGet-seed#[BA5BFFA382390CBC]) [ ] o.a.s.c.TestPullReplica tearDown deleting collection
[junit4] 2> 1533999 INFO (qtp12181658-15434) [n:127.0.0.1:40037_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=pull_replica_test_real_time_get&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1534002 INFO (OverseerThreadFactory-6136-thread-2-processing-n:127.0.0.1:57109_solr) [n:127.0.0.1:57109_solr ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
[junit4] 2> 1534002 INFO (OverseerCollectionConfigSetProcessor-98646220977405958-127.0.0.1:57109_solr-n_0000000000) [n:127.0.0.1:57109_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1534003 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_real_time_get.shard1.replica_n1, tag=null
[junit4] 2> 1534003 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_real_time_get.shard1.replica_n2, tag=null
[junit4] 2> 1534003 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@304ebaf4: rootName = solr_40037, domain = solr.core.pull_replica_test_real_time_get.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.pull_replica_test_real_time_get.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@5afb262c
[junit4] 2> 1534003 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@45a1d68e: rootName = solr_57109, domain = solr.core.pull_replica_test_real_time_get.shard1.replica_n2, service url = null, agent id = null] for registry solr.core.pull_replica_test_real_time_get.shard1.replica_n2 / com.codahale.metrics.MetricRegistry@4bfe5178
[junit4] 2> 1534025 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr ] o.a.s.c.SolrCore [pull_replica_test_real_time_get_shard1_replica_n2] CLOSING SolrCore org.apache.solr.core.SolrCore@62679b0d
[junit4] 2> 1534025 INFO (qtp441150007-15446) [n:127.0.0.1:57109_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_real_time_get.shard1.replica_p6, tag=null
[junit4] 2> 1534026 INFO (qtp441150007-15446) [n:127.0.0.1:57109_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6d8db3d4: rootName = solr_57109, domain = solr.core.pull_replica_test_real_time_get.shard1.replica_p6, service url = null, agent id = null] for registry solr.core.pull_replica_test_real_time_get.shard1.replica_p6 / com.codahale.metrics.MetricRegistry@186a9e46
[junit4] 2> 1534028 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_real_time_get.shard1.replica_p4, tag=null
[junit4] 2> 1534028 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3c7595f6: rootName = solr_40037, domain = solr.core.pull_replica_test_real_time_get.shard1.replica_p4, service url = null, agent id = null] for registry solr.core.pull_replica_test_real_time_get.shard1.replica_p4 / com.codahale.metrics.MetricRegistry@5ea2afac
[junit4] 2> 1534035 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr ] o.a.s.c.SolrCore [pull_replica_test_real_time_get_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@51103f00
[junit4] 2> 1534039 INFO (qtp441150007-15446) [n:127.0.0.1:57109_solr ] o.a.s.c.ZkController pull_replica_test_real_time_get_shard1_replica_p6 stopping background replication from leader
[junit4] 2> 1534039 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_real_time_get.shard1.replica_n2, tag=1650957069
[junit4] 2> 1534039 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.pull_replica_test_real_time_get.shard1.leader, tag=1650957069
[junit4] 2> 1534039 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 1534039 INFO (qtp441150007-15441) [n:127.0.0.1:57109_solr ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@64068bd7 commitCommandVersion:0
[junit4] 2> 1534039 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr ] o.a.s.c.ZkController pull_replica_test_real_time_get_shard1_replica_p4 stopping background replication from leader
[junit4] 2> 1534042 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_real_time_get.shard1.replica_n1, tag=1360019200
[junit4] 2> 1534043 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.pull_replica_test_real_time_get.shard1.leader, tag=1360019200
[junit4] 2> 1534043 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 1534043 INFO (qtp12181658-15437) [n:127.0.0.1:40037_solr ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@bf51745 commitCommandVersion:0
[junit4] 2> 1534047 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr ] o.a.s.c.SolrCore [pull_replica_test_real_time_get_shard1_replica_p4] CLOSING SolrCore org.apache.solr.core.SolrCore@33f53e93
[junit4] 2> 1534047 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.pull_replica_test_real_time_get.shard1.replica_p4, tag=871710355
[junit4] 2> 1534047 INFO (qtp12181658-15433) [n:127.0.0.1:40037_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.pull_replica_test_real_time_get.shard1.leader, tag=871710355
[junit4] 2> 1534049 INFO (qtp12181658-15433) [n:127.0.0.1:400
[...truncated too long message...]
ing : 127.0.0.1:40037_solr
[junit4] 2> 1678748 WARN (zkCallback-2586-thread-4-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 1678748 INFO (jetty-closer-2564-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@509276c3{/solr,null,UNAVAILABLE}
[junit4] 2> 1678750 ERROR (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1678750 INFO (SUITE-TestPullReplica-seed#[BA5BFFA382390CBC]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41972 41972
[junit4] 2> 1678795 WARN (indexFetcher-6418-thread-1) [n:127.0.0.1:40037_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.h.IndexFetcher Could not log failed replication details
[junit4] 2> org.apache.lucene.store.AlreadyClosedException: Already closed
[junit4] 2> at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:337)
[junit4] 2> at org.apache.solr.handler.ReplicationHandler.loadReplicationProperties(ReplicationHandler.java:1131)
[junit4] 2> at org.apache.solr.handler.IndexFetcher.logReplicationTimeAndConfFiles(IndexFetcher.java:769)
[junit4] 2> at org.apache.solr.handler.IndexFetcher.cleanup(IndexFetcher.java:677)
[junit4] 2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:659)
[junit4] 2> at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:332)
[junit4] 2> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:419)
[junit4] 2> at org.apache.solr.handler.ReplicationHandler.lambda$setupPolling$12(ReplicationHandler.java:1183)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1678796 INFO (indexFetcher-6418-thread-1) [n:127.0.0.1:40037_solr c:pull_replica_test_kill_pull_replica s:shard1 r:core_node4 x:pull_replica_test_kill_pull_replica_shard1_replica_p2] o.a.s.h.ReplicationHandler Ignoring exception matching No registered leader was found
[junit4] 2> 1678814 INFO (Thread-3929) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:41972 41972
[junit4] 2> 1678822 WARN (Thread-3929) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 18 /solr/configs/conf
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 4 /solr/security.json
[junit4] 2> 3 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 30 /solr/collections/pull_replica_test_create_delete/state.json
[junit4] 2> 22 /solr/collections/pull_replica_test_kill_pull_replica/state.json
[junit4] 2> 21 /solr/collections/pull_replica_test_add_remove_pull_replica/state.json
[junit4] 2> 20 /solr/collections/pull_replica_test_kill_leader/state.json
[junit4] 2> 20 /solr/collections/pull_replica_test_remove_all_writer_replicas/state.json
[junit4] 2> 14 /solr/collections/pull_replica_test_add_docs/state.json
[junit4] 2> 13 /solr/collections/pull_replica_test_real_time_get/state.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 2 /solr/overseer_elect/election/98646220977405958-127.0.0.1:57109_solr-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/collections
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 2 /solr/overseer/queue
[junit4] 2> 2 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplica_BA5BFFA382390CBC-001
[junit4] 2> Sep 12, 2017 1:13:47 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {foo=PostingsFormat(name=Asserting), foo_s=PostingsFormat(name=Asserting), id=PostingsFormat(name=MockRandom)}, docValues:{}, maxPointsInLeafNode=1547, maxMBSortInHeap=6.794535083992006, sim=RandomSimilarity(queryNorm=false): {}, locale=it-IT, timezone=Turkey
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=1,free=223970464,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [TestFieldSortValues, QueryEqualityTest, InfixSuggestersTest, TestSolrFieldCacheBean, MBeansHandlerTest, TestTrie, MoveReplicaTest, BaseCdcrDistributedZkTest, SampleTest, TestIntervalFaceting, TestDocTermOrds, TestAnalyzeInfixSuggestions, AssignTest, TestLeaderElectionWithEmptyReplica, OverriddenZkACLAndCredentialsProvidersTest, MoveReplicaHDFSTest, UpdateLogTest, MergeStrategyTest, ChaosMonkeySafeLeaderTest, TestRealTimeGet, TestEmbeddedSolrServerSchemaAPI, TestMaxScoreQueryParser, AnalyticsMergeStrategyTest, TestRandomFlRTGCloud, CoreSorterTest, MetricsConfigTest, HdfsBasicDistributedZkTest, PreAnalyzedFieldManagedSchemaCloudTest, TestLegacyField, TestSortByMinMaxFunction, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestSolrJ, TestSchemaResource, TestBM25SimilarityFactory, TestSolrDeletionPolicy1, TestLRUStatsCache, ZkNodePropsTest, TestUseDocValuesAsStored, TestQueryTypes, TestConfig, SpellingQueryConverterTest, PreAnalyzedFieldTest, TestPerFieldSimilarity, FileBasedSpellCheckerTest, DistanceUnitsTest, SuggesterFSTTest, BlockJoinFacetSimpleTest, SolrGraphiteReporterTest, DistributedTermsComponentTest, HdfsRecoveryZkTest, TestHdfsBackupRestoreCore, TestMinMaxOnMultiValuedField, TestClusterStateMutator, TestNumericTerms32, ExitableDirectoryReaderTest, ShufflingReplicaListTransformerTest, UtilsToolTest, TestDocumentBuilder, TestPushWriter, TestInfoStreamLogging, OpenExchangeRatesOrgProviderTest, BitVectorTest, TestScoreJoinQPNoScore, SaslZkACLProviderTest, OpenCloseCoreStressTest, TestRangeQuery, DistributedFacetPivotSmallTest, TestRTGBase, TestManagedSchema, TestTolerantUpdateProcessorRandomCloud, TestJoin, ShowFileRequestHandlerTest, CachingDirectoryFactoryTest, SpatialRPTFieldTypeTest, BigEndianAscendingWordDeserializerTest, ShardRoutingCustomTest, DistributedQueryElevationComponentTest, TestComplexPhraseQParserPlugin, SolrIndexConfigTest, TestGraphTermsQParserPlugin, ConnectionReuseTest, AnalysisErrorHandlingTest, DistributedQueueTest, TestGraphMLResponseWriter, TestDefaultStatsCache, ClassificationUpdateProcessorIntegrationTest, TestManagedResource, TestInPlaceUpdatesDistrib, BadIndexSchemaTest, JsonLoaderTest, TestFastOutputStream, DistributedQueryComponentCustomSortTest, TestOverriddenPrefixQueryForCustomFieldType, PingRequestHandlerTest, TestNumericRangeQuery64, TestConfigsApi, MigrateRouteKeyTest, TestUninvertingReader, TestCollapseQParserPlugin, StatsComponentTest, TestFileDictionaryLookup, BlobRepositoryCloudTest, BasicDistributedZk2Test, TestConfigOverlay, DOMUtilTest, TestPostingsSolrHighlighter, TestSearchPerf, UpdateParamsTest, ChangedSchemaMergeTest, TestRebalanceLeaders, FieldMutatingUpdateProcessorTest, BadCopyFieldTest, LoggingHandlerTest, TestPartialUpdateDeduplication, DistributedFacetPivotLongTailTest, TestDocSet, CollectionStateFormat2Test, HdfsChaosMonkeySafeLeaderTest, DistributedMLTComponentTest, TestSuggestSpellingConverter, TestRecovery, TestQueryUtils, MoreLikeThisHandlerTest, PreAnalyzedUpdateProcessorTest, PluginInfoTest, SuggestComponentContextFilterQueryTest, TestDynamicFieldResource, BasicAuthStandaloneTest, TestCloudManagedSchema, SimpleMLTQParserTest, TestPolicyCloud, TestSolrConfigHandler, TestCloudNestedDocsSort, CollectionsAPIAsyncDistributedZkTest, TestSolrCloudSnapshots, TestSolrConfigHandlerCloud, TestFieldCacheSortRandom, TestSlowCompositeReaderWrapper, TestClusterProperties, TestJsonFacetRefinement, TestSimpleQParserPlugin, TestOnReconnectListenerSupport, TestCopyFieldCollectionResource, TestUniqueKeyFieldResource, TemplateUpdateProcessorTest, TermsComponentTest, AtomicUpdatesTest, TestSolrCLIRunExample, TestSimpleTrackingShardHandler, DocumentBuilderTest, SolrRequestParserTest, TestRuleBasedAuthorizationPlugin, TestApiFramework, BlockDirectoryTest, OverseerModifyCollectionTest, DistributedDebugComponentTest, HdfsSyncSliceTest, TestNRTOpen, DistribDocExpirationUpdateProcessorTest, TestBadConfig, EnumFieldTest, SolrCloudExampleTest, HttpSolrCallGetCoreTest, TestHttpShardHandlerFactory, DistributedSuggestComponentTest, ZkFailoverTest, JSONWriterTest, NodeMutatorTest, TestDFISimilarityFactory, DistributedIntervalFacetingTest, TestStressLucene, TestFastLRUCache, SpellCheckCollatorTest, TestCollationField, BasicZkTest, OverseerStatusTest, DebugComponentTest, LeaderInitiatedRecoveryOnCommitTest, TestPullReplica]
[junit4] Completed [355/731 (1!)] on J1 in 149.68s, 10 tests, 1 failure, 1 skipped <<< FAILURES!
[...truncated 47466 lines...]
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 172 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/172/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest
Error Message:
Timeout waiting for all live and active
Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
at __randomizedtesting.SeedInfo.seed([F524A849B49803B9:7652F7BB62E10D18]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest(TestCloudRecovery.java:184)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12994 lines...]
[junit4] Suite: org.apache.solr.cloud.TestCloudRecovery
[junit4] 2> 3419055 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/init-core-data-001
[junit4] 2> 3419056 WARN (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=6 numCloses=6
[junit4] 2> 3419056 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 3419057 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 3419058 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001
[junit4] 2> 3419058 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 3419058 INFO (Thread-5307) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 3419058 INFO (Thread-5307) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 3419059 ERROR (Thread-5307) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 3419158 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:52005
[junit4] 2> 3419175 INFO (jetty-launcher-6119-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 3419175 INFO (jetty-launcher-6119-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 3419177 INFO (jetty-launcher-6119-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5f07733e{/solr,null,AVAILABLE}
[junit4] 2> 3419178 INFO (jetty-launcher-6119-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2b2c9500{/solr,null,AVAILABLE}
[junit4] 2> 3419178 INFO (jetty-launcher-6119-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@37d08d16{HTTP/1.1,[http/1.1]}{127.0.0.1:40710}
[junit4] 2> 3419180 INFO (jetty-launcher-6119-thread-1) [ ] o.e.j.s.Server Started @3422338ms
[junit4] 2> 3419180 INFO (jetty-launcher-6119-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40710}
[junit4] 2> 3419180 INFO (jetty-launcher-6119-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3b4fcfff{HTTP/1.1,[http/1.1]}{127.0.0.1:33195}
[junit4] 2> 3419181 INFO (jetty-launcher-6119-thread-2) [ ] o.e.j.s.Server Started @3422339ms
[junit4] 2> 3419181 ERROR (jetty-launcher-6119-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3419181 INFO (jetty-launcher-6119-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33195}
[junit4] 2> 3419184 ERROR (jetty-launcher-6119-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3419184 INFO (jetty-launcher-6119-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.1.0
[junit4] 2> 3419187 INFO (jetty-launcher-6119-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.1.0
[junit4] 2> 3419187 INFO (jetty-launcher-6119-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3419187 INFO (jetty-launcher-6119-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3419187 INFO (jetty-launcher-6119-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 3419187 INFO (jetty-launcher-6119-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 3419187 INFO (jetty-launcher-6119-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-12T05:14:15.430Z
[junit4] 2> 3419187 INFO (jetty-launcher-6119-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-12T05:14:15.430Z
[junit4] 2> 3419189 INFO (jetty-launcher-6119-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3419190 INFO (jetty-launcher-6119-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3419208 INFO (jetty-launcher-6119-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52005/solr
[junit4] 2> 3419208 INFO (jetty-launcher-6119-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52005/solr
[junit4] 2> 3419351 INFO (jetty-launcher-6119-thread-1) [n:127.0.0.1:40710_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3419352 INFO (jetty-launcher-6119-thread-1) [n:127.0.0.1:40710_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40710_solr
[junit4] 2> 3419353 INFO (jetty-launcher-6119-thread-1) [n:127.0.0.1:40710_solr ] o.a.s.c.Overseer Overseer (id=98644345179471878-127.0.0.1:40710_solr-n_0000000000) starting
[junit4] 2> 3419358 INFO (jetty-launcher-6119-thread-2) [n:127.0.0.1:33195_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3419359 INFO (jetty-launcher-6119-thread-2) [n:127.0.0.1:33195_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33195_solr
[junit4] 2> 3419363 INFO (zkCallback-6131-thread-1-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3419366 INFO (zkCallback-6130-thread-1-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3419368 INFO (jetty-launcher-6119-thread-1) [n:127.0.0.1:40710_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40710_solr
[junit4] 2> 3419370 INFO (zkCallback-6131-thread-1-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3419371 INFO (zkCallback-6130-thread-1-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3419422 INFO (jetty-launcher-6119-thread-2) [n:127.0.0.1:33195_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33195.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1baa46a2
[junit4] 2> 3419427 INFO (jetty-launcher-6119-thread-2) [n:127.0.0.1:33195_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33195.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1baa46a2
[junit4] 2> 3419427 INFO (jetty-launcher-6119-thread-2) [n:127.0.0.1:33195_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33195.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1baa46a2
[junit4] 2> 3419428 INFO (jetty-launcher-6119-thread-2) [n:127.0.0.1:33195_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001/node2/.
[junit4] 2> 3419474 INFO (jetty-launcher-6119-thread-1) [n:127.0.0.1:40710_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40710.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1baa46a2
[junit4] 2> 3419480 INFO (jetty-launcher-6119-thread-1) [n:127.0.0.1:40710_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40710.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1baa46a2
[junit4] 2> 3419480 INFO (jetty-launcher-6119-thread-1) [n:127.0.0.1:40710_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40710.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1baa46a2
[junit4] 2> 3419480 INFO (jetty-launcher-6119-thread-1) [n:127.0.0.1:40710_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001/node1/.
[junit4] 2> 3419502 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 3419503 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:52005/solr ready
[junit4] 2> 3419516 INFO (qtp1776366090-30035) [n:127.0.0.1:33195_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3419519 INFO (OverseerThreadFactory-11543-thread-1-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 3419631 INFO (OverseerStateUpdate-98644345179471878-127.0.0.1:40710_solr-n_0000000000) [n:127.0.0.1:40710_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_t1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:40710/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 3419635 INFO (OverseerStateUpdate-98644345179471878-127.0.0.1:40710_solr-n_0000000000) [n:127.0.0.1:40710_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_t2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:33195/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 3419642 INFO (OverseerStateUpdate-98644345179471878-127.0.0.1:40710_solr-n_0000000000) [n:127.0.0.1:40710_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_t4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:40710/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 3419645 INFO (OverseerStateUpdate-98644345179471878-127.0.0.1:40710_solr-n_0000000000) [n:127.0.0.1:40710_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_t6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:33195/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 3419854 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 3419854 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 3419854 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3419854 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 3419854 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 3419855 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3419968 INFO (zkCallback-6131-thread-1-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3419968 INFO (zkCallback-6131-thread-2-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3419968 INFO (zkCallback-6130-thread-1-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3419968 INFO (zkCallback-6130-thread-2-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3420882 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 3420883 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 3420885 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 3420891 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 3420897 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.IndexSchema [collection1_shard2_replica_t4] Schema name=minimal
[junit4] 2> 3420897 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.IndexSchema [collection1_shard1_replica_t1] Schema name=minimal
[junit4] 2> 3420897 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.IndexSchema [collection1_shard2_replica_t6] Schema name=minimal
[junit4] 2> 3420921 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3420921 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3420922 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t1' using configuration from collection collection1, trusted=true
[junit4] 2> 3420922 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t6' using configuration from collection collection1, trusted=true
[junit4] 2> 3420922 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33195.solr.core.collection1.shard2.replica_t6' (registry 'solr.core.collection1.shard2.replica_t6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1baa46a2
[junit4] 2> 3420922 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40710.solr.core.collection1.shard1.replica_t1' (registry 'solr.core.collection1.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1baa46a2
[junit4] 2> 3420922 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3420922 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3420922 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore [[collection1_shard2_replica_t6] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001/node2/collection1_shard2_replica_t6], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001/node2/./collection1_shard2_replica_t6/data/]
[junit4] 2> 3420922 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [[collection1_shard1_replica_t1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001/node1/collection1_shard1_replica_t1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001/node1/./collection1_shard1_replica_t1/data/]
[junit4] 2> 3420921 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3420924 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t4' using configuration from collection collection1, trusted=true
[junit4] 2> 3420924 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40710.solr.core.collection1.shard2.replica_t4' (registry 'solr.core.collection1.shard2.replica_t4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1baa46a2
[junit4] 2> 3420924 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3420924 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore [[collection1_shard2_replica_t4] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001/node1/collection1_shard2_replica_t4], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001/node1/./collection1_shard2_replica_t4/data/]
[junit4] 2> 3420930 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.IndexSchema [collection1_shard1_replica_t2] Schema name=minimal
[junit4] 2> 3420932 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3420932 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t2' using configuration from collection collection1, trusted=true
[junit4] 2> 3420932 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33195.solr.core.collection1.shard1.replica_t2' (registry 'solr.core.collection1.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1baa46a2
[junit4] 2> 3420932 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3420932 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore [[collection1_shard1_replica_t2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001/node2/collection1_shard1_replica_t2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001/tempDir-001/node2/./collection1_shard1_replica_t2/data/]
[junit4] 2> 3420982 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3420982 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3420982 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3420982 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3420984 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@30f8cc07[collection1_shard1_replica_t1] main]
[junit4] 2> 3420985 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 3420986 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 3420986 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 3420987 INFO (searcherExecutor-11548-thread-1-processing-n:127.0.0.1:40710_solr x:collection1_shard1_replica_t1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [collection1_shard1_replica_t1] Registered new searcher Searcher@30f8cc07[collection1_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3420987 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578309524893204480
[junit4] 2> 3420987 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3420987 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3420988 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3420988 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3420990 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.SolrIndexSearcher Opening [Searcher@2f7471da[collection1_shard2_replica_t4] main]
[junit4] 2> 3420991 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 3420992 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 3420993 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3420993 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3420993 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 3420994 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3420994 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3420995 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 3420995 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578309524901593088
[junit4] 2> 3420996 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.SolrIndexSearcher Opening [Searcher@1dc3240d[collection1_shard2_replica_t6] main]
[junit4] 2> 3420998 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 3420998 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 3420999 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 3420999 INFO (searcherExecutor-11550-thread-1-processing-n:127.0.0.1:40710_solr x:collection1_shard2_replica_t4 s:shard2 c:collection1 r:core_node7) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore [collection1_shard2_replica_t4] Registered new searcher Searcher@2f7471da[collection1_shard2_replica_t4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3421002 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
[junit4] 2> 3421003 INFO (searcherExecutor-11549-thread-1-processing-n:127.0.0.1:33195_solr x:collection1_shard2_replica_t6 s:shard2 c:collection1 r:core_node8) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@1dc3240d[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3421004 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578309524911030272
[junit4] 2> 3421012 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3421012 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3421013 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3421013 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3421015 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@6fde77bf[collection1_shard1_replica_t2] main]
[junit4] 2> 3421016 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 3421017 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 3421017 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 3421017 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578309524924661760
[junit4] 2> 3421018 INFO (searcherExecutor-11551-thread-1-processing-n:127.0.0.1:33195_solr x:collection1_shard1_replica_t2 s:shard1 c:collection1 r:core_node5) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore [collection1_shard1_replica_t2] Registered new searcher Searcher@6fde77bf[collection1_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3421105 INFO (zkCallback-6131-thread-2-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3421105 INFO (zkCallback-6130-thread-1-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3421105 INFO (zkCallback-6130-thread-2-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3421105 INFO (zkCallback-6131-thread-1-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3421495 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 3421495 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 3421495 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40710/solr/collection1_shard1_replica_t1/
[junit4] 2> 3421495 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_t1 url=http://127.0.0.1:40710/solr START replicas=[http://127.0.0.1:33195/solr/collection1_shard1_replica_t2/] nUpdates=100
[junit4] 2> 3421497 INFO (qtp1776366090-30031) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 3421504 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 3421504 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 3421504 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40710/solr/collection1_shard2_replica_t4/
[junit4] 2> 3421504 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_t4 url=http://127.0.0.1:40710/solr START replicas=[http://127.0.0.1:33195/solr/collection1_shard2_replica_t6/] nUpdates=100
[junit4] 2> 3421505 INFO (qtp1776366090-30037) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 3421797 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_t1 url=http://127.0.0.1:40710/solr DONE. We have no versions. sync failed.
[junit4] 2> 3421797 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 3421797 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 3421797 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 3421798 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ZkController collection1_shard1_replica_t1 stopping background replication from leader
[junit4] 2> 3421802 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40710/solr/collection1_shard1_replica_t1/ shard1
[junit4] 2> 3421805 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_t4 url=http://127.0.0.1:40710/solr DONE. We have no versions. sync failed.
[junit4] 2> 3421806 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 3421806 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 3421806 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 3421806 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ZkController collection1_shard2_replica_t4 stopping background replication from leader
[junit4] 2> 3421810 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40710/solr/collection1_shard2_replica_t4/ shard2
[junit4] 2> 3421914 INFO (zkCallback-6131-thread-1-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3421914 INFO (zkCallback-6131-thread-2-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3421916 INFO (zkCallback-6130-thread-1-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3421916 INFO (zkCallback-6130-thread-2-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3421954 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 3421956 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2102
[junit4] 2> 3421961 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 3421964 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2110
[junit4] 2> 3422009 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ZkController collection1_shard2_replica_t6 starting background replication from leader
[junit4] 2> 3422009 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 3422010 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 3422011 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Replica core_node7 is leader but it's state is down, skipping replication
[junit4] 2> 3422011 INFO (qtp1776366090-30039) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2157
[junit4] 2> 3422024 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ZkController collection1_shard1_replica_t2 starting background replication from leader
[junit4] 2> 3422024 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 3422025 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 3422027 INFO (qtp1776366090-30036) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2173
[junit4] 2> 3422031 INFO (qtp1776366090-30035) [n:127.0.0.1:33195_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 3422133 INFO (zkCallback-6131-thread-1-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3422133 INFO (zkCallback-6130-thread-1-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3422133 INFO (zkCallback-6131-thread-2-processing-n:127.0.0.1:40710_solr) [n:127.0.0.1:40710_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3422133 INFO (zkCallback-6130-thread-2-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 3423032 INFO (qtp1776366090-30035) [n:127.0.0.1:33195_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=0&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=2&wt=javabin&version=2} status=0 QTime=3517
[junit4] 2> 3423032 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4] 2> 3423033 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 3423312 INFO (TEST-TestCloudRecovery.leaderRecoverFromLogOnStartupTest-seed#[F524A849B49803B9]) [ ] o.a.s.SolrTestCaseJ4 ###Starting leaderRecoverFromLogOnStartupTest
[junit4] 2> 3423328 INFO (qtp1776366090-30040) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t6] webapp=/solr path=/update params={update.distrib=FROMLEADER&_version_=-1578309527337435136&distrib.from=http://127.0.0.1:40710/solr/collection1_shard2_replica_t4/&wt=javabin&version=2}{deleteByQuery=*:* (-1578309527337435136)} 0 0
[junit4] 2> 3423329 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t4] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:40710/solr/collection1_shard1_replica_t1/&wt=javabin&version=2}{deleteByQuery=*:* (-1578309527337435136)} 0 11
[junit4] 2> 3423330 INFO (qtp1776366090-30038) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t2] webapp=/solr path=/update params={update.distrib=FROMLEADER&_version_=-1578309527335337984&distrib.from=http://127.0.0.1:40710/solr/collection1_shard1_replica_t1/&wt=javabin&version=2}{deleteByQuery=*:* (-1578309527335337984)} 0 3
[junit4] 2> 3423330 INFO (qtp1536975560-30030) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t1] webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1578309527335337984)} 0 15
[junit4] 2> 3423332 INFO (qtp1776366090-30031) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.TestInjection Start waiting for replica in sync with leader
[junit4] 2> 3423333 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1578309527353163776,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 3423333 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 3423333 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1578309527353163776,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 3423333 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 3423334 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 3423334 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:40710/solr/collection1_shard1_replica_t1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 3423334 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 3423334 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t4] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:40710/solr/collection1_shard1_replica_t1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
[junit4] 2> 3423334 INFO (qtp1776366090-30037) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.TestInjection Start waiting for replica in sync with leader
[junit4] 2> 3423336 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3423338 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3423529 INFO (OverseerCollectionConfigSetProcessor-98644345179471878-127.0.0.1:40710_solr-n_0000000000) [n:127.0.0.1:40710_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 3423680 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Updated masterUrl to http://127.0.0.1:40710/solr/collection1_shard1_replica_t1/
[junit4] 2> 3423681 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 3423681 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 3423682 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 3423682 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 3423682 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 3423842 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3423868 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3424345 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3424372 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3424848 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3424875 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3425010 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
[junit4] 2> 3425010 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Updated masterUrl to http://127.0.0.1:40710/solr/collection1_shard2_replica_t4/
[junit4] 2> 3425011 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 3425011 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 3425011 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 3425011 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 3425011 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 3425011 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 3425012 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 3425012 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@11c32944 commitCommandVersion:0
[junit4] 2> 3425017 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.SolrIndexSearcher Opening [Searcher@1b3b4234[collection1_shard2_replica_t6] main]
[junit4] 2> 3425017 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 3425018 INFO (searcherExecutor-11549-thread-1-processing-n:127.0.0.1:33195_solr x:collection1_shard2_replica_t6 s:shard2 c:collection1 r:core_node8) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@1b3b4234[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3425351 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3425381 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3425854 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3425884 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3426357 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3426388 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3426679 INFO (qtp1536975560-30030) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 3426679 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 3426679 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 3426679 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 3426679 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 3426860 INFO (qtp1536975560-30030) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3426891 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3427364 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3427395 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3427868 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3427900 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3428009 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 3428010 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 3428010 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 3428010 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 3428010 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's version: 1505193261255
[junit4] 2> 3428380 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3428403 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3428884 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3428908 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3429390 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3429413 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3429679 INFO (qtp1536975560-30030) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 3429679 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 3429679 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 3429679 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 3429679 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 3429896 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3429917 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3430405 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3430423 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3430911 INFO (qtp1536975560-30030) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3430929 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3431008 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 3431009 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 3431009 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 3431009 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 3431009 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's version: 1505193261255
[junit4] 2> 3431416 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3431432 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3431918 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3431936 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3432422 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3432440 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3432677 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 3432677 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 3432677 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 3432677 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 3432677 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 3432929 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3432942 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3433435 INFO (qtp1536975560-30030) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3433446 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3433943 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3433950 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3434007 INFO (qtp1536975560-30030) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 3434007 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 3434007 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 3434007 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 3434007 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's version: 1505193261255
[junit4] 2> 3434449 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3434453 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3434955 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3434956 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3435458 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3435459 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3435677 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 3435677 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 3435677 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 3435677 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 3435677 INFO (indexFetcher-11571-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 3435961 INFO (qtp1536975560-30041) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3435962 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3436465 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3436465 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3436967 INFO (qtp1536975560-30028) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3436967 INFO (qtp1536975560-30026) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3437006 INFO (qtp1536975560-30029) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 3437006 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 3437006 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 3437006 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 3437006 INFO (indexFetcher-11569-thread-1) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's version: 1505193261255
[junit4] 2> 3437471 INFO (qtp1536975560-30032) [n:127.0.0.1:40710_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3437471 INFO (qtp1536975560-30030) [n:127.0.0.1:40710_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 3437979 INFO (qtp1536975560-30041) [n:127
[...truncated too long message...]
c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=859216293
[junit4] 2> 3590540 WARN (recoveryExecutor-6154-thread-1-processing-n:127.0.0.1:33195_solr x:collection1_shard2_replica_t6 s:shard2 c:collection1 r:core_node8) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.RecoveryStrategy Stopping recovery for core=[collection1_shard2_replica_t6] coreNodeName=[core_node8]
[junit4] 2> 3590546 WARN (updateExecutor-6153-thread-1-processing-n:127.0.0.1:33195_solr x:collection1_shard2_replica_t6 s:shard2 c:collection1 r:core_node8) [n:127.0.0.1:33195_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.DefaultSolrCoreState Skipping recovery because Solr is shutdown
[junit4] 2> 3590547 INFO (jetty-closer-6120-thread-1) [ ] o.a.s.c.Overseer Overseer (id=98644345179471889-127.0.0.1:33195_solr-n_0000000004) closing
[junit4] 2> 3590548 INFO (OverseerStateUpdate-98644345179471889-127.0.0.1:33195_solr-n_0000000004) [n:127.0.0.1:33195_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33195_solr
[junit4] 2> 3590552 WARN (zkCallback-6156-thread-2-processing-n:127.0.0.1:33195_solr) [n:127.0.0.1:33195_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 3590553 INFO (jetty-closer-6120-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@9ddf2f1{/solr,null,UNAVAILABLE}
[junit4] 2> 3590556 ERROR (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 3590557 INFO (SUITE-TestCloudRecovery-seed#[F524A849B49803B9]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52005 52005
[junit4] 2> 3590620 INFO (Thread-5307) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:52005 52005
[junit4] 2> 3590629 WARN (Thread-5307) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 7 /solr/aliases.json
[junit4] 2> 7 /solr/clusterprops.json
[junit4] 2> 6 /solr/security.json
[junit4] 2> 6 /solr/configs/config
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 128 /solr/collections/collection1/state.json
[junit4] 2> 7 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 7 /solr/collections
[junit4] 2> 4 /solr/overseer/queue
[junit4] 2> 4 /solr/overseer/collection-queue-work
[junit4] 2> 3 /solr/live_nodes
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_F524A849B49803B9-001
[junit4] 2> Sep 12, 2017 5:17:06 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=true): {}, locale=ru, timezone=US/Pacific
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=1,free=132720584,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [HttpSolrCallGetCoreTest, TestComponentsName, AutoCommitTest, RequestLoggingTest, TestUniqueKeyFieldResource, TestPartialUpdateDeduplication, MoveReplicaTest, DistributedIntervalFacetingTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestCoreDiscovery, TestFieldSortValues, NodeMutatorTest, SynonymTokenizerTest, TestUseDocValuesAsStored, DistributedFacetPivotLongTailTest, TestSolrCloudWithSecureImpersonation, FileUtilsTest, TestInPlaceUpdatesDistrib, TestManagedSchemaThreadSafety, PeerSyncReplicationTest, TestGraphTermsQParserPlugin, TestMaxScoreQueryParser, MoreLikeThisHandlerTest, TermVectorComponentDistributedTest, TestExportWriter, SolrJmxReporterTest, CoreSorterTest, DeleteReplicaTest, DeleteLastCustomShardedReplicaTest, TolerantUpdateProcessorTest, TestMinMaxOnMultiValuedField, TestFieldTypeResource, TestInfoStreamLogging, TestBadConfig, SolrCloudReportersTest, TestConfigSetsAPIExclusivity, CollectionsAPIDistributedZkTest, CustomCollectionTest, TestSolrCloudSnapshots, TestShardHandlerFactory, BadCopyFieldTest, TermsComponentTest, ClassificationUpdateProcessorIntegrationTest, HdfsNNFailoverTest, TestZkChroot, TestRebalanceLeaders, OverseerStatusTest, TestSQLHandler, BasicZkTest, LeaderInitiatedRecoveryOnCommitTest, TestRTGBase, SpatialRPTFieldTypeTest, ZkSolrClientTest, ExitableDirectoryReaderTest, MetricUtilsTest, TestMultiValuedNumericRangeQuery, DistributedMLTComponentTest, TestComplexPhraseQParserPlugin, TestStressUserVersions, OpenExchangeRatesOrgProviderTest, TestPerFieldSimilarity, TestCollectionsAPIViaSolrCloudCluster, FileBasedSpellCheckerTest, ReplicationFactorTest, TestSearchPerf, SortSpecParsingTest, TestConfig, TestDistributedMissingSort, UpdateLogTest, OverriddenZkACLAndCredentialsProvidersTest, TestClusterProperties, ExternalFileFieldSortTest, TestCryptoKeys, TestApiFramework, PropertiesRequestHandlerTest, TestRealTimeGet, SolrCloudExampleTest, TestBM25SimilarityFactory, HdfsBasicDistributedZkTest, LeaderElectionIntegrationTest, DistributedDebugComponentTest, TestGraphMLResponseWriter, TestNumericRangeQuery64, ConcurrentDeleteAndCreateCollectionTest, TestAuthorizationFramework, SpellingQueryConverterTest, LeaderFailoverAfterPartitionTest, SolrMetricManagerTest, CloneFieldUpdateProcessorFactoryTest, SystemInfoHandlerTest, MetricsConfigTest, TestJsonFacetRefinement, TestExclusionRuleCollectionAccess, TestConfigSetProperties, TestMultiWordSynonyms, UtilsToolTest, TestPullReplica, ShufflingReplicaListTransformerTest, TermVectorComponentTest, TestInitQParser, TestPerFieldSimilarityWithDefaultOverride, TestReloadDeadlock, DataDrivenBlockJoinTest, TestLegacyNumericUtils, TestValueSourceCache, TestClusterStateMutator, CopyFieldTest, QueryResultKeyTest, TestReqParamsAPI, CachingDirectoryFactoryTest, TestMergePolicyConfig, OverseerModifyCollectionTest, TestSubQueryTransformerCrossCore, SimpleCollectionCreateDeleteTest, TestPayloadScoreQParserPlugin, HttpPartitionTest, TestIndexSearcher, TestQueryTypes, UpdateParamsTest, TestMacroExpander, IndexSchemaRuntimeFieldTest, TestXmlQParser, SolrRequestParserTest, InfixSuggestersTest, MigrateRouteKeyTest, TestHttpShardHandlerFactory, TestEmbeddedSolrServerSchemaAPI, TestStressInPlaceUpdates, DOMUtilTest, TestCloudManagedSchema, SolrCoreCheckLockOnStartupTest, AliasIntegrationTest, StatelessScriptUpdateProcessorFactoryTest, ConnectionReuseTest, DocValuesMultiTest, TestManagedSchema, RollingRestartTest, TestTlogReplica, TestCloudNestedDocsSort, TestSolrConfigHandlerCloud, MoveReplicaHDFSTest, TestPostingsSolrHighlighter, SolrMetricsIntegrationTest, BadIndexSchemaTest, BigEndianAscendingWordDeserializerTest, TestStressLucene, ChangedSchemaMergeTest, TestFileDictionaryLookup, UpdateRequestProcessorFactoryTest, SearchHandlerTest, EnumFieldTest, TestCollationField, TestRandomFaceting, ZkNodePropsTest, ConnectionManagerTest, TestEmbeddedSolrServerAdminHandler, SolrIndexMetricsTest, HighlighterMaxOffsetTest, TestCSVLoader, HdfsSyncSliceTest, TestRandomCollapseQParserPlugin, TestPushWriter, SolrPluginUtilsTest, TestFieldResource, TemplateUpdateProcessorTest, AssignTest, DeleteStatusTest, BasicAuthStandaloneTest, BlockJoinFacetDistribTest, ConvertedLegacyTest, DocumentBuilderTest, SignatureUpdateProcessorFactoryTest, BasicFunctionalityTest, HdfsChaosMonkeySafeLeaderTest, ShardRoutingCustomTest, CursorMarkTest, TestSSLRandomization, TestCrossCoreJoin, ChaosMonkeySafeLeaderTest, TestOnReconnectListenerSupport, BasicDistributedZk2Test, UnloadDistributedZkTest, OpenCloseCoreStressTest, ShardRoutingTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, LeaderElectionTest, TestRandomDVFaceting, ZkCLITest, TestDistributedSearch, TestDistributedGrouping, TestFaceting, TestRecovery, DistributedSpellCheckComponentTest, TestStressReorder, TestJoin, TestReload, DistributedTermsComponentTest, TestRangeQuery, SolrCoreTest, SpellCheckComponentTest, SolrCmdDistributorTest, TestSort, TestFiltering, SoftAutoCommitTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, JsonLoaderTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, CSVRequestHandlerTest, SOLR749Test, TestSolrIndexConfig, TestQuerySenderNoQuery, SolrIndexConfigTest, BadComponentTest, SampleTest, TestDocSet, NumericFieldsTest, TestElisionMultitermQuery, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TimeZoneUtilsTest, TestFastOutputStream, URLClassifyProcessorTest, TestFastLRUCache, PreAnalyzedFieldTest, TestSolrJ, TestLRUCache, CircularListTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestSimpleTrackingShardHandler, BaseCdcrDistributedZkTest, CdcrBootstrapTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionStateFormat2Test, CollectionsAPIAsyncDistributedZkTest, ConfigSetsAPITest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest, DistribCursorPagingTest, DistribJoinFromCollectionTest, DocValuesNotIndexedTest, LeaderFailureAfterFreshStartTest, OverseerCollectionConfigSetProcessorTest, OverseerRolesTest, OverseerTaskQueueTest, RecoveryAfterSoftCommitTest, TestCloudRecovery]
[junit4] Completed [542/731 (1!)] on J1 in 171.89s, 2 tests, 1 failure <<< FAILURES!
[...truncated 46850 lines...]
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 171 -
Still unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/171/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC
7 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.MoveReplicaHDFSTest
Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64369 within 30000 ms
Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64369 within 30000 ms
at __randomizedtesting.SeedInfo.seed([6A10D1E01D2864EB]:0)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:183)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:112)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:99)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForAllNodes(MiniSolrCloudCluster.java:268)
at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:262)
at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:190)
at org.apache.solr.cloud.MoveReplicaTest.setupCluster(MoveReplicaTest.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 127.0.0.1:64369 within 30000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:233)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:175)
... 31 more
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.MoveReplicaHDFSTest
Error Message:
Stack Trace:
java.lang.NullPointerException
at __randomizedtesting.SeedInfo.seed([6A10D1E01D2864EB]:0)
at org.apache.solr.cloud.MoveReplicaHDFSTest.teardownClass(MoveReplicaHDFSTest.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:897)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.MoveReplicaHDFSTest
Error Message:
54 threads leaked from SUITE scope at org.apache.solr.cloud.MoveReplicaHDFSTest: 1) Thread[id=22670, name=qtp1468910649-22670, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=22703, name=qtp550049991-22703, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=22667, name=qtp1468910649-22667, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=22702, name=org.eclipse.jetty.server.session.HashSessionManager@3933cd4dTimer, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=22694, name=qtp781667853-22694, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=22682, name=Scheduler-1418452177, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=22696, name=qtp781667853-22696, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=22706, name=qtp781667853-22706, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 9) Thread[id=22664, name=org.eclipse.jetty.server.session.HashSessionManager@22debd8aTimer, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=22710, name=qtp1468910649-22710, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 11) Thread[id=22652, name=jetty-launcher-3360-thread-1, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 12) Thread[id=22659, name=qtp1514945000-22659, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 13) Thread[id=22671, name=qtp1468910649-22671-acceptor-0@772231b4-ServerConnector@15c588a4{HTTP/1.1,[http/1.1]}{127.0.0.1:46663}, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 14) Thread[id=22663, name=qtp1514945000-22663, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 15) Thread[id=22647, name=SyncThread:0, state=WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) 16) Thread[id=22661, name=qtp1514945000-22661, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 17) Thread[id=22691, name=org.eclipse.jetty.server.session.HashSessionManager@5d825de9Timer, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18) Thread[id=22673, name=org.eclipse.jetty.server.session.HashSessionManager@5345d214Timer, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19) Thread[id=22654, name=jetty-launcher-3360-thread-3, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20) Thread[id=22695, name=qtp781667853-22695-acceptor-0@12e191cb-ServerConnector@1ee90fd9{HTTP/1.1,[http/1.1]}{127.0.0.1:45062}, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 21) Thread[id=22700, name=qtp781667853-22700, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 22) Thread[id=22704, name=qtp550049991-22704, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 23) Thread[id=22657, name=qtp1514945000-22657, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 24) Thread[id=22660, name=qtp1514945000-22660, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 25) Thread[id=22648, name=ProcessThread(sid:0 cport:64369):, state=WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122) 26) Thread[id=22677, name=qtp1514945000-22677, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 27) Thread[id=22658, name=qtp1514945000-22658-acceptor-0@4fc9e5d8-ServerConnector@42bffcf4{HTTP/1.1,[http/1.1]}{127.0.0.1:64369}, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 28) Thread[id=22668, name=qtp1468910649-22668, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 29) Thread[id=22653, name=jetty-launcher-3360-thread-2, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 30) Thread[id=22655, name=jetty-launcher-3360-thread-4, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 31) Thread[id=22662, name=qtp1514945000-22662, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 32) Thread[id=22685, name=qtp550049991-22685, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 33) Thread[id=22687, name=qtp550049991-22687, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 34) Thread[id=22705, name=qtp781667853-22705, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 35) Thread[id=22676, name=qtp1468910649-22676, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 36) Thread[id=22690, name=qtp550049991-22690, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 37) Thread[id=22683, name=qtp550049991-22683, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 38) Thread[id=22646, name=SessionTracker, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at java.lang.Object.wait(Native Method) at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) 39) Thread[id=22644, name=Thread-4658, state=WAITING, group=TGRP-MoveReplicaHDFSTest] at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297) at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309) at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490) 40) Thread[id=22701, name=qtp781667853-22701, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 41) Thread[id=22666, name=qtp1468910649-22666, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 42) Thread[id=22697, name=qtp781667853-22697, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 43) Thread[id=22645, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173) at java.lang.Thread.run(Thread.java:748) 44) Thread[id=22672, name=qtp1468910649-22672, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 45) Thread[id=22669, name=qtp1468910649-22669, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 46) Thread[id=22688, name=qtp550049991-22688, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 47) Thread[id=22684, name=qtp550049991-22684, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 48) Thread[id=22686, name=qtp550049991-22686-acceptor-0@5fd3672a-ServerConnector@17a19a24{HTTP/1.1,[http/1.1]}{127.0.0.1:56636}, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 49) Thread[id=22711, name=qtp781667853-22711, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 50) Thread[id=22681, name=qtp1514945000-22681, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 51) Thread[id=22712, name=qtp550049991-22712, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 52) Thread[id=22656, name=qtp1514945000-22656, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 53) Thread[id=22699, name=qtp781667853-22699, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 54) Thread[id=22707, name=qtp1468910649-22707, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 54 threads leaked from SUITE scope at org.apache.solr.cloud.MoveReplicaHDFSTest:
1) Thread[id=22670, name=qtp1468910649-22670, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=22703, name=qtp550049991-22703, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=22667, name=qtp1468910649-22667, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=22702, name=org.eclipse.jetty.server.session.HashSessionManager@3933cd4dTimer, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=22694, name=qtp781667853-22694, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
6) Thread[id=22682, name=Scheduler-1418452177, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
7) Thread[id=22696, name=qtp781667853-22696, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest]
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
8) Thread[id=22706, name=qtp781667853-22706, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
9) Thread[id=22664, name=org.eclipse.jetty.server.session.HashSessionManager@22debd8aTimer, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
10) Thread[id=22710, name=qtp1468910649-22710, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
11) Thread[id=22652, name=jetty-launcher-3360-thread-1, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
12) Thread[id=22659, name=qtp1514945000-22659, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
13) Thread[id=22671, name=qtp1468910649-22671-acceptor-0@772231b4-ServerConnector@15c588a4{HTTP/1.1,[http/1.1]}{127.0.0.1:46663}, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
14) Thread[id=22663, name=qtp1514945000-22663, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
15) Thread[id=22647, name=SyncThread:0, state=WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127)
16) Thread[id=22661, name=qtp1514945000-22661, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
17) Thread[id=22691, name=org.eclipse.jetty.server.session.HashSessionManager@5d825de9Timer, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
18) Thread[id=22673, name=org.eclipse.jetty.server.session.HashSessionManager@5345d214Timer, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
19) Thread[id=22654, name=jetty-launcher-3360-thread-3, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
20) Thread[id=22695, name=qtp781667853-22695-acceptor-0@12e191cb-ServerConnector@1ee90fd9{HTTP/1.1,[http/1.1]}{127.0.0.1:45062}, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
21) Thread[id=22700, name=qtp781667853-22700, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
22) Thread[id=22704, name=qtp550049991-22704, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
23) Thread[id=22657, name=qtp1514945000-22657, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
24) Thread[id=22660, name=qtp1514945000-22660, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
25) Thread[id=22648, name=ProcessThread(sid:0 cport:64369):, state=WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:122)
26) Thread[id=22677, name=qtp1514945000-22677, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest]
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
27) Thread[id=22658, name=qtp1514945000-22658-acceptor-0@4fc9e5d8-ServerConnector@42bffcf4{HTTP/1.1,[http/1.1]}{127.0.0.1:64369}, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
28) Thread[id=22668, name=qtp1468910649-22668, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
29) Thread[id=22653, name=jetty-launcher-3360-thread-2, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
30) Thread[id=22655, name=jetty-launcher-3360-thread-4, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
31) Thread[id=22662, name=qtp1514945000-22662, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
32) Thread[id=22685, name=qtp550049991-22685, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
33) Thread[id=22687, name=qtp550049991-22687, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
34) Thread[id=22705, name=qtp781667853-22705, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
35) Thread[id=22676, name=qtp1468910649-22676, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
36) Thread[id=22690, name=qtp550049991-22690, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
37) Thread[id=22683, name=qtp550049991-22683, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest]
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
38) Thread[id=22646, name=SessionTracker, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at java.lang.Object.wait(Native Method)
at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
39) Thread[id=22644, name=Thread-4658, state=WAITING, group=TGRP-MoveReplicaHDFSTest]
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
at java.lang.Thread.join(Thread.java:1326)
at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:297)
at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:309)
at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:490)
40) Thread[id=22701, name=qtp781667853-22701, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
41) Thread[id=22666, name=qtp1468910649-22666, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest]
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
42) Thread[id=22697, name=qtp781667853-22697, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
43) Thread[id=22645, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest]
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:173)
at java.lang.Thread.run(Thread.java:748)
44) Thread[id=22672, name=qtp1468910649-22672, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
45) Thread[id=22669, name=qtp1468910649-22669, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
46) Thread[id=22688, name=qtp550049991-22688, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
47) Thread[id=22684, name=qtp550049991-22684, state=TIMED_WAITING, group=TGRP-MoveReplicaHDFSTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
48) Thread[id=22686, name=qtp550049991-22686-acceptor-0@5fd3672a-ServerConnector@17a19a24{HTTP/1.1,[http/1.1]}{127.0.0.1:56636}, state=RUNNABLE, group=TGRP-MoveReplicaHDFSTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept
[...truncated too long message...]
lection1_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 225376 INFO (zkCallback-986-thread-3-processing-n:127.0.0.1:45284_solr) [n:127.0.0.1:45284_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/workQueue/state.json] for collection [workQueue] has occurred - updating... (live nodes size: [1])
[junit4] 2> 225376 INFO (zkCallback-986-thread-2-processing-n:127.0.0.1:45284_solr) [n:127.0.0.1:45284_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/workQueue/state.json] for collection [workQueue] has occurred - updating... (live nodes size: [1])
[junit4] 2> 225390 INFO (coreCloseExecutor-647-thread-4) [n:127.0.0.1:45284_solr c:destination s:shard1 r:core_node3 x:destination_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.destination.shard1.leader, tag=868584935
[junit4] 2> 225390 INFO (coreCloseExecutor-647-thread-3) [n:127.0.0.1:45284_solr c:mainCorpus s:shard1 r:core_node2 x:mainCorpus_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.mainCorpus.shard1.replica_n1, tag=186700374
[junit4] 2> 225390 INFO (coreCloseExecutor-647-thread-3) [n:127.0.0.1:45284_solr c:mainCorpus s:shard1 r:core_node2 x:mainCorpus_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@771adef6: rootName = solr_45284, domain = solr.core.mainCorpus.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.mainCorpus.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@2c22ec5
[junit4] 2> 225407 INFO (coreCloseExecutor-647-thread-3) [n:127.0.0.1:45284_solr c:mainCorpus s:shard1 r:core_node2 x:mainCorpus_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.mainCorpus.shard1.leader, tag=186700374
[junit4] 2> 225453 INFO (jetty-closer-973-thread-4) [ ] o.a.s.c.Overseer Overseer (id=98642552625233926-127.0.0.1:45284_solr-n_0000000000) closing
[junit4] 2> 225453 INFO (OverseerStateUpdate-98642552625233926-127.0.0.1:45284_solr-n_0000000000) [n:127.0.0.1:45284_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:45284_solr
[junit4] 2> 226639 WARN (zkCallback-993-thread-1-processing-n:127.0.0.1:48597_solr) [n:127.0.0.1:48597_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 226640 INFO (jetty-closer-973-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@232eb1bf{/solr,null,UNAVAILABLE}
[junit4] 2> 226747 WARN (zkCallback-995-thread-3-processing-n:127.0.0.1:53858_solr) [n:127.0.0.1:53858_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 226748 INFO (jetty-closer-973-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2cce7de0{/solr,null,UNAVAILABLE}
[junit4] 2> 226760 WARN (zkCallback-994-thread-2-processing-n:127.0.0.1:40780_solr) [n:127.0.0.1:40780_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 226761 INFO (jetty-closer-973-thread-3) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@44c5d4ff{/solr,null,UNAVAILABLE}
[junit4] 2> 226966 WARN (zkCallback-986-thread-3-processing-n:127.0.0.1:45284_solr) [n:127.0.0.1:45284_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 226967 INFO (jetty-closer-973-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@762839b2{/solr,null,UNAVAILABLE}
[junit4] 2> 226970 ERROR (SUITE-StreamExpressionTest-seed#[C3D7DA26982934D]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 226973 INFO (SUITE-StreamExpressionTest-seed#[C3D7DA26982934D]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43764 43764
[junit4] 2> 238541 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
[junit4] 2> java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:182)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 238542 INFO (Thread-355) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43764 43764
[junit4] 2> 238543 WARN (Thread-355) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 72 /solr/aliases.json
[junit4] 2> 72 /solr/clusterprops.json
[junit4] 2> 8 /solr/configs/conf
[junit4] 2> 4 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 72 /solr/clusterstate.json
[junit4] 2> 21 /solr/collections/destination/state.json
[junit4] 2> 21 /solr/collections/mainCorpus/state.json
[junit4] 2> 20 /solr/collections/parallelDestinationCollection/state.json
[junit4] 2> 20 /solr/collections/destinationCollection/state.json
[junit4] 2> 19 /solr/collections/workQueue/state.json
[junit4] 2> 18 /solr/collections/parallelDestinationCollection1/state.json
[junit4] 2> 11 /solr/collections/collection1_collection/state.json
[junit4] 2> 2 /solr/overseer_elect/election/98642552625233932-127.0.0.1:53858_solr-n_0000000001
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 72 /solr/live_nodes
[junit4] 2> 72 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.io.stream.StreamExpressionTest_C3D7DA26982934D-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {expr_s=FST50, join2_s=PostingsFormat(name=LuceneFixedGap), multiDefault=PostingsFormat(name=LuceneFixedGap), subject=PostingsFormat(name=LuceneFixedGap), a1_s=PostingsFormat(name=LuceneFixedGap), a2_s=PostingsFormat(name=Memory), test_t=PostingsFormat(name=Memory), level1_s=FST50, body_t=PostingsFormat(name=Memory), ident_s=PostingsFormat(name=LuceneFixedGap), s_multi=PostingsFormat(name=Memory), col_s=PostingsFormat(name=LuceneFixedGap), a_ss=PostingsFormat(name=Direct), level2_s=PostingsFormat(name=Direct), a_s=PostingsFormat(name=Memory), a_t=PostingsFormat(name=Direct), side_s=PostingsFormat(name=Memory), id=FST50, text=PostingsFormat(name=Memory)}, docValues:{expr_s=DocValuesFormat(name=Memory), join2_s=DocValuesFormat(name=Lucene70), multiDefault=DocValuesFormat(name=Lucene70), price_f=DocValuesFormat(name=Lucene70), a1_s=DocValuesFormat(name=Lucene70), intDefault=DocValuesFormat(name=Asserting), a_f=DocValuesFormat(name=Lucene70), join1_i=DocValuesFormat(name=Memory), a2_s=DocValuesFormat(name=Direct), a_i=DocValuesFormat(name=Memory), join3_i=DocValuesFormat(name=Direct), test_i=DocValuesFormat(name=Lucene70), field_i=DocValuesFormat(name=Lucene70), level1_s=DocValuesFormat(name=Memory), s_multi=DocValuesFormat(name=Direct), col_s=DocValuesFormat(name=Lucene70), a_ss=DocValuesFormat(name=Asserting), level2_s=DocValuesFormat(name=Asserting), a_s=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Asserting), i_multi=DocValuesFormat(name=Memory), b_ls=DocValuesFormat(name=Lucene70), test_dt=DocValuesFormat(name=Memory), ident_s=DocValuesFormat(name=Lucene70), order_i=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Asserting), side_s=DocValuesFormat(name=Direct), miles_i=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1790, maxMBSortInHeap=7.518650305943785, sim=RandomSimilarity(queryNorm=true): {}, locale=sk-SK, timezone=Asia/Amman
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_141 (64-bit)/cpus=3,threads=1,free=148206912,total=339738624
[junit4] 2> NOTE: All tests run in this JVM: [SolrExceptionTest, MergeIndexesEmbeddedTest, NotEvaluatorTest, TestLBHttpSolrClient, NoOpResponseParserTest, LargeVolumeBinaryJettyTest, TestDelegationTokenResponse, JdbcTest, HyperbolicSineEvaluatorTest, StreamingTest, ShardParamsTest, CloudSolrClientBuilderTest, AddEvaluatorTest, TestRetryUtil, NormalDistributionEvaluatorTest, RecursiveEvaluatorTest, CeilingEvaluatorTest, CloudSolrClientMultiConstructorTest, LessThanEvaluatorTest, HttpSolrClientSSLAuthConPoolTest, ConcurrentUpdateSolrClientBuilderTest, ArcSineEvaluatorTest, TestCollectionAdminRequest, TestNamedListCodec, TestPolicy, SelectWithEvaluatorsTest, ExclusiveOrEvaluatorTest, SolrQueryTest, TestJsonRecordReader, ArrayEvaluatorTest, TestCloudSolrClientConnections, GreaterThanEvaluatorTest, StreamExpressionToExpessionTest, HttpSolrClientBuilderTest, ClientUtilsTest, SolrExampleStreamingBinaryTest, SolrExampleXMLTest, SolrExampleEmbeddedTest, TestSolrProperties, TestFastInputStream, DocumentAnalysisResponseTest, TestUpdateRequestCodec, FacetFieldTest, CloudSolrClientCacheTest, HttpSolrClientConPoolTest, GraphTest, JdbcDriverTest, StreamExpressionTest]
[junit4] Completed [142/145 (1!)] on J1 in 107.01s, 109 tests, 1 error, 9 skipped <<< FAILURES!
[...truncated 45366 lines...]