You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2015/12/26 14:49:38 UTC

[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 893 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/893/

1 tests failed.
FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test10MinutesOld

Error Message:
Captured an uncaught exception in thread: Thread[id=15567, name=Thread-10543, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=15567, name=Thread-10543, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded




Build Log:
[...truncated 10289 lines...]
   [junit4] Suite: org.apache.solr.core.OpenCloseCoreStressTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/init-core-data-001
   [junit4]   2> 856952 INFO  (SUITE-OpenCloseCoreStressTest-seed#[F2D6F15CCEE2DB10]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true)
   [junit4] IGNOR/A 0.00s J1 | OpenCloseCoreStressTest.test1Hour
   [junit4]    > Assumption #1: 'weekly' test group is disabled (@Weekly())
   [junit4]   2> 856957 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test15SecondsOld
   [junit4]   2> 856973 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 856974 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7725ef2b{/solr,null,AVAILABLE}
   [junit4]   2> 856976 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.e.j.s.ServerConnector Started ServerConnector@3a2abc6d{HTTP/1.1,[http/1.1]}{127.0.0.1:59245}
   [junit4]   2> 856976 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.e.j.s.Server Started @861140ms
   [junit4]   2> 856976 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59245}
   [junit4]   2> 856976 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 856976 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001'
   [junit4]   2> 856977 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 856977 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 856977 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/solr.xml
   [junit4]   2> 856982 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001
   [junit4]   2> 856982 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreContainer New CoreContainer 444121789
   [junit4]   2> 856982 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001]
   [junit4]   2> 856982 WARN  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/lib
   [junit4]   2> 856982 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 856984 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 856984 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 856984 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 856984 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 856984 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 856985 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001
   [junit4]   2> 856986 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=00000_core, config=solrconfig-minimal.xml, loadOnStartup=false, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 856986 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CorePropertiesLocator Found core 00000_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core
   [junit4]   2> 856986 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=00004_core, config=solrconfig-minimal.xml, loadOnStartup=false, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 856987 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CorePropertiesLocator Found core 00004_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core
   [junit4]   2> 856987 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=00003_core, config=solrconfig-minimal.xml, loadOnStartup=false, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 856987 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CorePropertiesLocator Found core 00003_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core
   [junit4]   2> 856988 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=00002_core, config=solrconfig-minimal.xml, loadOnStartup=false, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 856988 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CorePropertiesLocator Found core 00002_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core
   [junit4]   2> 856988 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=00006_core, config=solrconfig-minimal.xml, loadOnStartup=false, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 856989 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CorePropertiesLocator Found core 00006_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core
   [junit4]   2> 856989 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=00001_core, config=solrconfig-minimal.xml, loadOnStartup=false, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 856989 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CorePropertiesLocator Found core 00001_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core
   [junit4]   2> 856990 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=00005_core, config=solrconfig-minimal.xml, loadOnStartup=false, schema=schema-tiny.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 856990 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CorePropertiesLocator Found core 00005_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core
   [junit4]   2> 856990 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.CorePropertiesLocator Found 7 core definitions
   [junit4]   2> 856990 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 856990 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 856998 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.OpenCloseCoreStressTest Starting indexing and querying
   [junit4]   2> 856999 INFO  (TEST-OpenCloseCoreStressTest.test15SecondsOld-seed#[F2D6F15CCEE2DB10]) [    ] o.a.s.c.OpenCloseCoreStressTest 
   [junit4]   2> 
   [junit4]   2> 
   [junit4]   2> 
   [junit4]   2> 
   [junit4]   2> Starting a 15 second cycle, seconds left: 15. Seconds run so far: 0.
   [junit4]   2> 857004 INFO  (Thread-10236) [    ] o.a.s.c.OneIndexer Starting indexing thread: 15188
   [junit4]   2> 857008 INFO  (Thread-10237) [    ] o.a.s.c.OneIndexer Starting indexing thread: 15189
   [junit4]   2> 857012 INFO  (Thread-10238) [    ] o.a.s.c.OneIndexer Starting indexing thread: 15191
   [junit4]   2> 857017 INFO  (qtp103958222-15175) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core'
   [junit4]   2> 857017 INFO  (qtp103958222-15175) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857017 INFO  (qtp103958222-15175) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857020 INFO  (Thread-10240) [    ] o.a.s.c.OneIndexer Starting indexing thread: 15193
   [junit4]   2> 857020 INFO  (Thread-10241) [    ] o.a.s.c.OneIndexer Starting indexing thread: 15194
   [junit4]   2> 857025 INFO  (qtp103958222-15175) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 857028 INFO  (Thread-10239) [    ] o.a.s.c.OneIndexer Starting indexing thread: 15192
   [junit4]   2> 857030 INFO  (qtp103958222-15175) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 857030 INFO  (Thread-10242) [    ] o.a.s.c.OneIndexer Starting indexing thread: 15195
   [junit4]   2> 857031 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core'
   [junit4]   2> 857031 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857031 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857031 INFO  (Thread-10243) [    ] o.a.s.c.OneIndexer Starting indexing thread: 15196
   [junit4]   2> 857033 INFO  (qtp103958222-15179) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core'
   [junit4]   2> 857033 INFO  (qtp103958222-15179) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857033 INFO  (qtp103958222-15179) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857033 INFO  (qtp103958222-15180) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core'
   [junit4]   2> 857033 INFO  (qtp103958222-15180) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857033 INFO  (qtp103958222-15180) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857035 INFO  (qtp103958222-15175) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857035 INFO  (Thread-10244) [    ] o.a.s.c.OneIndexer Starting indexing thread: 15197
   [junit4]   2> 857036 INFO  (Thread-10245) [    ] o.a.s.c.OneQuery Starting query thread: 15198
   [junit4]   2> 857037 INFO  (Thread-10246) [    ] o.a.s.c.OneQuery Starting query thread: 15199
   [junit4]   2> 857037 INFO  (Thread-10247) [    ] o.a.s.c.OneQuery Starting query thread: 15200
   [junit4]   2> 857037 INFO  (Thread-10248) [    ] o.a.s.c.OneQuery Starting query thread: 15201
   [junit4]   2> 857037 INFO  (qtp103958222-15175) [    ] o.a.s.s.IndexSchema [00006_core] Schema name=tiny
   [junit4]   2> 857038 INFO  (Thread-10249) [    ] o.a.s.c.OneQuery Starting query thread: 15202
   [junit4]   2> 857038 INFO  (Thread-10250) [    ] o.a.s.c.OneQuery Starting query thread: 15203
   [junit4]   2> 857038 INFO  (Thread-10251) [    ] o.a.s.c.OneQuery Starting query thread: 15204
   [junit4]   2> 857041 INFO  (Thread-10252) [    ] o.a.s.c.OneQuery Starting query thread: 15205
   [junit4]   2> 857041 INFO  (qtp103958222-15175) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 857042 INFO  (qtp103958222-15180) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 857043 INFO  (qtp103958222-15175) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core'
   [junit4]   2> 857043 INFO  (qtp103958222-15175) [    ] o.a.s.c.CoreContainer Creating SolrCore '00006_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core
   [junit4]   2> 857044 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857044 INFO  (qtp103958222-15179) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 857044 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 857044 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrCore [[00006_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core/data/]
   [junit4]   2> 857044 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrCore JMX monitoring not detected for core: 00006_core
   [junit4]   2> 857046 INFO  (qtp103958222-15179) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 857046 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 857046 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core/data
   [junit4]   2> 857047 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core/data/index/
   [junit4]   2> 857047 WARN  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrCore [00006_core] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857048 INFO  (qtp103958222-15180) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 857048 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core/data/index
   [junit4]   2> 857048 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@202064e3
   [junit4]   2> 857048 INFO  (Thread-10253) [    ] o.a.s.c.OneQuery Starting query thread: 15206
   [junit4]   2> 857049 INFO  (qtp103958222-15182) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core'
   [junit4]   2> 857049 INFO  (qtp103958222-15182) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857049 INFO  (qtp103958222-15182) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857050 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@6368b4a4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@53295188),segFN=segments_1,generation=1}
   [junit4]   2> 857050 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857052 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857053 INFO  (qtp103958222-15180) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857054 INFO  (qtp103958222-15176) [    ] o.a.s.s.IndexSchema [00001_core] Schema name=tiny
   [junit4]   2> 857055 INFO  (qtp103958222-15182) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 857056 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857056 INFO  (qtp103958222-15180) [    ] o.a.s.s.IndexSchema [00000_core] Schema name=tiny
   [junit4]   2> 857057 INFO  (qtp103958222-15176) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 857057 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857070 INFO  (OldIndexDirectoryCleanupThreadForCore-00006_core) [    x:00006_core] o.a.s.c.SolrCore Looking for old index directories to cleanup for core 00006_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core/data/
   [junit4]   2> 857071 WARN  (OldIndexDirectoryCleanupThreadForCore-00006_core) [    x:00006_core] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 857071 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857071 INFO  (qtp103958222-15176) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core'
   [junit4]   2> 857071 INFO  (qtp103958222-15176) [    ] o.a.s.c.CoreContainer Creating SolrCore '00001_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core
   [junit4]   2> 857071 INFO  (qtp103958222-15180) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 857071 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857072 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrCore [[00001_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core/data/]
   [junit4]   2> 857072 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrCore JMX monitoring not detected for core: 00001_core
   [junit4]   2> 857072 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857073 INFO  (qtp103958222-15180) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core'
   [junit4]   2> 857073 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857073 INFO  (qtp103958222-15180) [    ] o.a.s.c.CoreContainer Creating SolrCore '00000_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core
   [junit4]   2> 857070 INFO  (qtp103958222-15182) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 857073 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857073 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrCore [[00000_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core/data/]
   [junit4]   2> 857073 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrCore JMX monitoring not detected for core: 00000_core
   [junit4]   2> 857074 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 857075 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 857075 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core/data
   [junit4]   2> 857076 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core/data/index/
   [junit4]   2> 857076 WARN  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrCore [00001_core] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857076 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core/data
   [junit4]   2> 857076 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core/data/index
   [junit4]   2> 857076 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core/data/index/
   [junit4]   2> 857076 WARN  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrCore [00000_core] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857076 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@50580e45
   [junit4]   2> 857077 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 857077 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core/data/index
   [junit4]   2> 857077 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 857077 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@1b2dfca7
   [junit4]   2> 857078 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@21531412 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@300b805),segFN=segments_1,generation=1}
   [junit4]   2> 857078 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857078 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=41.0859375, floorSegmentMB=0.3466796875, forceMergeDeletesPctAllowed=0.20317280994581743, segmentsPerTier=15.0, maxCFSSegmentSizeMB=0.4736328125, noCFSRatio=1.0
   [junit4]   2> 857078 INFO  (qtp103958222-15182) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857078 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@6368b4a4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@53295188),segFN=segments_1,generation=1}
   [junit4]   2> 857079 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857079 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.s.SolrIndexSearcher Opening Searcher@a8ed128[00006_core] main
   [junit4]   2> 857080 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core/conf
   [junit4]   2> 857080 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core/conf}
   [junit4]   2> 857080 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00006_core/conf
   [junit4]   2> 857080 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 857080 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 857080 INFO  (OldIndexDirectoryCleanupThreadForCore-00001_core) [    x:00001_core] o.a.s.c.SolrCore Looking for old index directories to cleanup for core 00001_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core/data/
   [junit4]   2> 857081 WARN  (OldIndexDirectoryCleanupThreadForCore-00001_core) [    x:00001_core] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 857081 INFO  (qtp103958222-15182) [    ] o.a.s.s.IndexSchema [00005_core] Schema name=tiny
   [junit4]   2> 857081 INFO  (searcherExecutor-3685-thread-1-processing-x:00006_core) [    x:00006_core] o.a.s.c.SolrCore [00006_core] Registered new searcher Searcher@a8ed128[00006_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 857081 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.SolrCores Opening transient core 00006_core
   [junit4]   2> 857081 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.c.CoreContainer registering core: 00006_core
   [junit4]   2> 857082 INFO  (qtp103958222-15179) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857085 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@27c9ab48 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@49bd3d76),segFN=segments_1,generation=1}
   [junit4]   2> 857086 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857089 INFO  (qtp103958222-15179) [    ] o.a.s.s.IndexSchema [00004_core] Schema name=tiny
   [junit4]   2> 857089 INFO  (qtp103958222-15182) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 857090 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857090 INFO  (qtp103958222-15182) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core'
   [junit4]   2> 857090 INFO  (qtp103958222-15182) [    ] o.a.s.c.CoreContainer Creating SolrCore '00005_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core
   [junit4]   2> 857090 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857090 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrCore [[00005_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core/data/]
   [junit4]   2> 857090 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrCore JMX monitoring not detected for core: 00005_core
   [junit4]   2> 857091 INFO  (qtp103958222-15179) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 857091 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857092 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core/data
   [junit4]   2> 857092 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857092 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core/data/index/
   [junit4]   2> 857092 WARN  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrCore [00005_core] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857092 INFO  (qtp103958222-15179) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core'
   [junit4]   2> 857092 INFO  (qtp103958222-15179) [    ] o.a.s.c.CoreContainer Creating SolrCore '00004_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core
   [junit4]   2> 857093 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core/data/index
   [junit4]   2> 857093 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857093 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@1f047de1
   [junit4]   2> 857093 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrCore [[00004_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core/data/]
   [junit4]   2> 857093 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrCore JMX monitoring not detected for core: 00004_core
   [junit4]   2> 857093 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857093 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@4dfdde07 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3ce19307),segFN=segments_1,generation=1}
   [junit4]   2> 857093 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857094 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857094 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core/data
   [junit4]   2> 857094 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core/data/index/
   [junit4]   2> 857094 WARN  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrCore [00004_core] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857095 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core/data/index
   [junit4]   2> 857095 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@75874e2f
   [junit4]   2> 857097 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 857098 INFO  (OldIndexDirectoryCleanupThreadForCore-00000_core) [    x:00000_core] o.a.s.c.SolrCore Looking for old index directories to cleanup for core 00000_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core/data/
   [junit4]   2> 857098 INFO  (qtp103958222-15210) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id9]} 0 26
   [junit4]   2> 857098 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@13094447 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@39ac24eb),segFN=segments_1,generation=1}
   [junit4]   2> 857098 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857098 WARN  (OldIndexDirectoryCleanupThreadForCore-00000_core) [    x:00000_core] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 857098 INFO  (qtp103958222-15178) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id1]} 0 78
   [junit4]   2> 857098 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id3]} 0 81
   [junit4]   2> 857100 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857101 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857102 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857102 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857103 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857103 INFO  (OldIndexDirectoryCleanupThreadForCore-00005_core) [    x:00005_core] o.a.s.c.SolrCore Looking for old index directories to cleanup for core 00005_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core/data/
   [junit4]   2> 857098 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 857103 WARN  (OldIndexDirectoryCleanupThreadForCore-00005_core) [    x:00005_core] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 857104 INFO  (OldIndexDirectoryCleanupThreadForCore-00004_core) [    x:00004_core] o.a.s.c.SolrCore Looking for old index directories to cleanup for core 00004_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core/data/
   [junit4]   2> 857104 WARN  (OldIndexDirectoryCleanupThreadForCore-00004_core) [    x:00004_core] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 857104 INFO  (qtp103958222-15213) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core'
   [junit4]   2> 857104 INFO  (qtp103958222-15213) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857104 INFO  (qtp103958222-15213) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857105 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 857105 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 857105 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 857105 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 857106 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=41.0859375, floorSegmentMB=0.3466796875, forceMergeDeletesPctAllowed=0.20317280994581743, segmentsPerTier=15.0, maxCFSSegmentSizeMB=0.4736328125, noCFSRatio=1.0
   [junit4]   2> 857106 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@4dfdde07 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3ce19307),segFN=segments_1,generation=1}
   [junit4]   2> 857106 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857107 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.s.SolrIndexSearcher Opening Searcher@69ac9619[00005_core] main
   [junit4]   2> 857107 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857107 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core/conf
   [junit4]   2> 857107 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core/conf}
   [junit4]   2> 857107 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00005_core/conf
   [junit4]   2> 857107 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857107 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 857107 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 857107 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 857108 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 857108 INFO  (searcherExecutor-3691-thread-1-processing-x:00005_core) [    x:00005_core] o.a.s.c.SolrCore [00005_core] Registered new searcher Searcher@69ac9619[00005_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 857108 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857108 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857108 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.SolrCores Opening transient core 00005_core
   [junit4]   2> 857108 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.CoreContainer registering core: 00005_core
   [junit4]   2> 857109 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857109 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857110 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=41.0859375, floorSegmentMB=0.3466796875, forceMergeDeletesPctAllowed=0.20317280994581743, segmentsPerTier=15.0, maxCFSSegmentSizeMB=0.4736328125, noCFSRatio=1.0
   [junit4]   2> 857110 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@21531412 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@300b805),segFN=segments_1,generation=1}
   [junit4]   2> 857110 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857110 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857110 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.s.SolrIndexSearcher Opening Searcher@70f9ec7a[00001_core] main
   [junit4]   2> 857110 INFO  (qtp103958222-15213) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 857111 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core/conf
   [junit4]   2> 857111 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core/conf}
   [junit4]   2> 857111 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857111 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00001_core/conf
   [junit4]   2> 857111 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 857111 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 857112 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.SolrCores Opening transient core 00001_core
   [junit4]   2> 857112 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.c.CoreContainer registering core: 00001_core
   [junit4]   2> 857112 INFO  (searcherExecutor-3686-thread-1-processing-x:00001_core) [    x:00001_core] o.a.s.c.SolrCore [00001_core] Registered new searcher Searcher@70f9ec7a[00001_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 857113 INFO  (qtp103958222-15213) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 857114 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857114 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id5]} 0 65
   [junit4]   2> 857114 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857116 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 857116 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 857116 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 857116 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 857117 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 857117 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 857117 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 857117 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 857121 INFO  (qtp103958222-15211) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id10]} 0 17
   [junit4]   2> 857121 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=41.0859375, floorSegmentMB=0.3466796875, forceMergeDeletesPctAllowed=0.20317280994581743, segmentsPerTier=15.0, maxCFSSegmentSizeMB=0.4736328125, noCFSRatio=1.0
   [junit4]   2> 857122 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@27c9ab48 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@49bd3d76),segFN=segments_1,generation=1}
   [junit4]   2> 857122 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857122 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.s.SolrIndexSearcher Opening Searcher@6bdd2721[00000_core] main
   [junit4]   2> 857122 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core/conf
   [junit4]   2> 857122 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core/conf}
   [junit4]   2> 857122 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00000_core/conf
   [junit4]   2> 857122 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 857123 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 857123 INFO  (searcherExecutor-3687-thread-1-processing-x:00000_core) [    x:00000_core] o.a.s.c.SolrCore [00000_core] Registered new searcher Searcher@6bdd2721[00000_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 857123 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.SolrCores Opening transient core 00000_core
   [junit4]   2> 857123 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.c.CoreContainer registering core: 00000_core
   [junit4]   2> 857123 INFO  (qtp103958222-15210) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id14]} 0 0
   [junit4]   2> 857126 INFO  (qtp103958222-15207) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id8]} 0 55
   [junit4]   2> 857126 INFO  (qtp103958222-15176) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id4]} 0 95
   [junit4]   2> 857127 INFO  (qtp103958222-15213) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857127 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id7]} 0 93
   [junit4]   2> 857127 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=41.0859375, floorSegmentMB=0.3466796875, forceMergeDeletesPctAllowed=0.20317280994581743, segmentsPerTier=15.0, maxCFSSegmentSizeMB=0.4736328125, noCFSRatio=1.0
   [junit4]   2> 857128 INFO  (qtp103958222-15209) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id12]} 0 24
   [junit4]   2> 857129 INFO  (qtp103958222-15210) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id17]} 0 0
   [junit4]   2> 857129 INFO  (qtp103958222-15207) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id16]} 0 0
   [junit4]   2> 857130 INFO  (qtp103958222-15210) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id15]} 0 1
   [junit4]   2> 857131 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core'
   [junit4]   2> 857131 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@13094447 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@39ac24eb),segFN=segments_1,generation=1}
   [junit4]   2> 857131 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857131 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 857131 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 857131 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.s.SolrIndexSearcher Opening Searcher@4e01c71a[00004_core] main
   [junit4]   2> 857132 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core/conf
   [junit4]   2> 857132 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core/conf}
   [junit4]   2> 857132 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00004_core/conf
   [junit4]   2> 857132 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 857132 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id22]} 0 0
   [junit4]   2> 857134 INFO  (qtp103958222-15213) [    ] o.a.s.s.IndexSchema [00002_core] Schema name=tiny
   [junit4]   2> 857134 INFO  (qtp103958222-15182) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id23]} 0 0
   [junit4]   2> 857132 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 857135 INFO  (searcherExecutor-3692-thread-1-processing-x:00004_core) [    x:00004_core] o.a.s.c.SolrCore [00004_core] Registered new searcher Searcher@4e01c71a[00004_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 857135 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.SolrCores Opening transient core 00004_core
   [junit4]   2> 857135 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.c.CoreContainer registering core: 00004_core
   [junit4]   2> 857138 INFO  (qtp103958222-15175) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id18]} 0 9
   [junit4]   2> 857139 INFO  (qtp103958222-15210) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id24]} 0 0
   [junit4]   2> 857140 INFO  (qtp103958222-15211) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id25]} 0 0
   [junit4]   2> 857141 INFO  (qtp103958222-15211) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 857143 INFO  (qtp103958222-15175) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 857143 INFO  (qtp103958222-15211) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id27]} 0 0
   [junit4]   2> 857143 INFO  (qtp103958222-15175) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 857148 INFO  (qtp103958222-15181) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id2]} 0 100
   [junit4]   2> 857149 INFO  (qtp103958222-15178) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id13]} 0 32
   [junit4]   2> 857150 INFO  (qtp103958222-15182) [    x:00005_core] o.a.s.c.S.Request [00005_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 857151 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id6]} 0 118
   [junit4]   2> 857153 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 857155 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 857158 INFO  (qtp103958222-15213) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 857159 INFO  (qtp103958222-15213) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core'
   [junit4]   2> 857159 INFO  (qtp103958222-15213) [    ] o.a.s.c.CoreContainer Creating SolrCore '00002_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core
   [junit4]   2> 857159 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857160 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrCore [[00002_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core/data/]
   [junit4]   2> 857160 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrCore JMX monitoring not detected for core: 00002_core
   [junit4]   2> 857161 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core/data
   [junit4]   2> 857161 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core/data/index/
   [junit4]   2> 857161 WARN  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrCore [00002_core] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857162 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core/data/index
   [junit4]   2> 857162 INFO  (qtp103958222-15176) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig-minimal.xml
   [junit4]   2> 857162 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2e12edcc
   [junit4]   2> 857173 INFO  (qtp103958222-15175) [    x:00005_core] o.a.s.c.S.Request [00005_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 857173 INFO  (qtp103958222-15178) [    x:00001_core] o.a.s.c.S.Request [00001_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 857174 INFO  (qtp103958222-15178) [    x:00004_core] o.a.s.c.S.Request [00004_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 857174 INFO  (qtp103958222-15175) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id29]} 0 0
   [junit4]   2> 857174 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@2ae2aefe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35fb9b7a),segFN=segments_1,generation=1}
   [junit4]   2> 857175 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857175 INFO  (OldIndexDirectoryCleanupThreadForCore-00002_core) [    x:00002_core] o.a.s.c.SolrCore Looking for old index directories to cleanup for core 00002_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core/data/
   [junit4]   2> 857175 WARN  (OldIndexDirectoryCleanupThreadForCore-00002_core) [    x:00002_core] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 857177 INFO  (qtp103958222-15176) [    ] o.a.s.s.IndexSchema [00003_core] Schema name=tiny
   [junit4]   2> 857177 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857178 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857179 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857180 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857180 INFO  (qtp103958222-15176) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 857181 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857182 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 857182 INFO  (qtp103958222-15176) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core'
   [junit4]   2> 857182 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 857183 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 857183 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 857183 INFO  (qtp103958222-15176) [    ] o.a.s.c.CoreContainer Creating SolrCore '00003_core' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core
   [junit4]   2> 857183 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=33, maxMergedSegmentMB=41.0859375, floorSegmentMB=0.3466796875, forceMergeDeletesPctAllowed=0.20317280994581743, segmentsPerTier=15.0, maxCFSSegmentSizeMB=0.4736328125, noCFSRatio=1.0
   [junit4]   2> 857183 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 857183 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@2ae2aefe lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@35fb9b7a),segFN=segments_1,generation=1}
   [junit4]   2> 857183 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857184 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.c.SolrCore [[00003_core] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core/data/]
   [junit4]   2> 857184 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.s.SolrIndexSearcher Opening Searcher@780df2c3[00002_core] main
   [junit4]   2> 857184 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.c.SolrCore JMX monitoring not detected for core: 00003_core
   [junit4]   2> 857184 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core/conf
   [junit4]   2> 857184 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core/conf}
   [junit4]   2> 857184 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00002_core/conf
   [junit4]   2> 857184 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 857184 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 857185 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.SolrCores Opening transient core 00002_core
   [junit4]   2> 857185 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core/data
   [junit4]   2> 857185 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.c.CoreContainer registering core: 00002_core
   [junit4]   2> 857185 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core/data/index/
   [junit4]   2> 857185 WARN  (qtp103958222-15176) [    x:00003_core] o.a.s.c.SolrCore [00003_core] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core/data/index' doesn't exist. Creating new index...
   [junit4]   2> 857185 INFO  (searcherExecutor-3705-thread-1-processing-x:00002_core) [    x:00002_core] o.a.s.c.SolrCore [00002_core] Registered new searcher Searcher@780df2c3[00002_core] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 857185 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core/data/index
   [junit4]   2> 857187 INFO  (qtp103958222-15210) [    x:00002_core] o.a.s.c.S.Request [00002_core]  webapp=/solr path=/select params={q=*:*&qt=/select&wt=javabin&version=2} hits=0 status=0 QTime=14
   [junit4]   2> 857188 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=7, maxMergeAtOnceExplicit=4, maxMergedSegmentMB=84.240234375, floorSegmentMB=2.0625, forceMergeDeletesPctAllowed=16.486488324252196, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8596981312086202
   [junit4]   2> 857188 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@39416cc3 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@311bccc4),segFN=segments_1,generation=1}
   [junit4]   2> 857188 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 857189 INFO  (qtp103958222-15175) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id32]} 0 13
   [junit4]   2> 857191 INFO  (qtp103958222-15179) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id33]} 0 0
   [junit4]   2> 857192 INFO  (qtp103958222-15213) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id11]} 0 88
   [junit4]   2> 857192 INFO  (qtp103958222-15180) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id26]} 0 49
   [junit4]   2> 857193 INFO  (qtp103958222-15207) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id20]} 0 61
   [junit4]   2> 857193 INFO  (OldIndexDirectoryCleanupThreadForCore-00003_core) [    x:00003_core] o.a.s.c.SolrCore Looking for old index directories to cleanup for core 00003_core in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core/data/
   [junit4]   2> 857193 WARN  (OldIndexDirectoryCleanupThreadForCore-00003_core) [    x:00003_core] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001/tempDir-001/00003_core/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 857194 INFO  (qtp103958222-15235) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id34]} 0 0
   [junit4]   2> 857196 INFO  (qtp103958222-15210) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id35]} 0 0
   [junit4]   2> 857196 INFO  (qtp103958222-15179) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id36]} 0 0
   [junit4]   2> 857196 INFO  (qtp103958222-15182) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id31]} 0 22
   [junit4]   2> 857195 INFO  (qtp103958222-15175) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id37]} 0 0
   [junit4]   2> 857203 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id42]} 0 2
   [junit4]   2> 857204 INFO  (qtp103958222-15235) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id41]} 0 2
   [junit4]   2> 857204 INFO  (qtp103958222-15210) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id38]} 0 2
   [junit4]   2> 857204 INFO  (qtp103958222-15207) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id40]} 0 3
   [junit4]   2> 857207 INFO  (qtp103958222-15180) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id43]} 0 1
   [junit4]   2> 857208 INFO  (qtp103958222-15175) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id45]} 0 0
   [junit4]   2> 857209 INFO  (qtp103958222-15179) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id47]} 0 0
   [junit4]   2> 857209 INFO  (qtp103958222-15182) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id44]} 0 3
   [junit4]   2> 857210 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id46]} 0 2
   [junit4]   2> 857211 INFO  (qtp103958222-15235) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id49]} 0 0
   [junit4]   2> 857212 INFO  (qtp103958222-15235) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id51]} 0 0
   [junit4]   2> 857212 INFO  (qtp103958222-15207) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id48]} 0 0
   [junit4]   2> 857213 INFO  (qtp103958222-15175) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id52]} 0 0
   [junit4]   2> 857213 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 857214 INFO  (qtp103958222-15179) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id53]} 0 0
   [junit4]   2> 857214 INFO  (qtp103958222-15182) [    x:00001_core] o.a.s.u.p.LogUpdateProcessorFactory [00001_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id54]} 0 0
   [junit4]   2> 857215 INFO  (qtp103958222-15180) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id55]} 0 0
   [junit4]   2> 857216 INFO  (qtp103958222-15235) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id56]} 0 0
   [junit4]   2> 857216 INFO  (qtp103958222-15207) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id57]} 0 0
   [junit4]   2> 857217 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857217 INFO  (qtp103958222-15175) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id58]} 0 0
   [junit4]   2> 857217 INFO  (qtp103958222-15179) [    x:00004_core] o.a.s.u.p.LogUpdateProcessorFactory [00004_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id59]} 0 0
   [junit4]   2> 857218 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id60]} 0 0
   [junit4]   2> 857219 INFO  (qtp103958222-15175) [    x:00006_core] o.a.s.u.p.LogUpdateProcessorFactory [00006_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id61]} 0 0
   [junit4]   2> 857219 INFO  (qtp103958222-15235) [    x:00000_core] o.a.s.u.p.LogUpdateProcessorFactory [00000_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id62]} 0 0
   [junit4]   2> 857220 INFO  (qtp103958222-15207) [    x:00005_core] o.a.s.u.p.LogUpdateProcessorFactory [00005_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id63]} 0 0
   [junit4]   2> 857221 INFO  (qtp103958222-15179) [    x:00002_core] o.a.s.u.p.LogUpdateProcessorFactory [00002_core]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[id65]} 0 0
   [junit4]   2> 857221 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 857222 INFO  (qtp103958222-15176) [    x:00003_core] o.a.s.h.l.XMLLoader xsltCacheLifetimeSecond

[...truncated too long message...]

4] ERROR    351s J1 | OpenCloseCoreStressTest.test10MinutesOld <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Core 00000_core bad! expected:<120701> but was:<120662>
   [junit4]    > 	at org.apache.solr.core.OpenCloseCoreStressTest.checkResults(OpenCloseCoreStressTest.java:298)
   [junit4]    > 	at org.apache.solr.core.OpenCloseCoreStressTest.doStress(OpenCloseCoreStressTest.java:197)
   [junit4]    > 	at org.apache.solr.core.OpenCloseCoreStressTest.test10MinutesOld(OpenCloseCoreStressTest.java:129)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=15567, name=Thread-10543, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceededThrowable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=15753, name=Lucene Merge Thread #27, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F2D6F15CCEE2DB10]:0)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]    > Caused by: org.apache.lucene.store.AlreadyClosedException: refusing to delete any files: this IndexWriter hit an unrecoverable exception
   [junit4]    > 	at org.apache.lucene.index.IndexFileDeleter.ensureOpen(IndexFileDeleter.java:347)
   [junit4]    > 	at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:731)
   [junit4]    > 	at org.apache.lucene.index.IndexFileDeleter.deletePendingFiles(IndexFileDeleter.java:520)
   [junit4]    > 	at org.apache.lucene.index.IndexFileDeleter.deleteNewFiles(IndexFileDeleter.java:723)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.deleteNewFiles(IndexWriter.java:4666)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4140)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3642)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceededThrowable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=15760, name=Lucene Merge Thread #28, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F2D6F15CCEE2DB10]:0)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceededThrowable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=15747, name=qtp2121050119-15747, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceededThrowable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=15766, name=Lucene Merge Thread #33, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F2D6F15CCEE2DB10]:0)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]    > Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot merge
   [junit4]    > 	at org.apache.lucene.index.IndexWriter._mergeInit(IndexWriter.java:3799)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeInit(IndexWriter.java:3779)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3633)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceededThrowable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=15755, name=Lucene Merge Thread #32, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F2D6F15CCEE2DB10]:0)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]   2> 1546848 INFO  (SUITE-OpenCloseCoreStressTest-seed#[F2D6F15CCEE2DB10]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.core.OpenCloseCoreStressTest_F2D6F15CCEE2DB10-001
   [junit4]   2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=de_CH, timezone=Pacific/Midway
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=4,threads=1,free=61790640,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [TestFreeTextSuggestions, TestManagedSynonymFilterFactory, ZkControllerTest, TestSolrDeletionPolicy2, HLLSerializationTest, SolrInfoMBeanTest, SolrIndexSplitterTest, ChaosMonkeySafeLeaderTest, TestFiltering, DistributedIntervalFacetingTest, TestBinaryResponseWriter, InfoHandlerTest, TestCloudManagedSchemaConcurrent, TestSortingResponseWriter, CoreAdminCreateDiscoverTest, TestJsonRequest, TestJettySolrRunner, TestRebalanceLeaders, BitVectorTest, TestTrie, TestWordDelimiterFilterFactory, TestUpdate, TestPerFieldSimilarityWithDefaultOverride, TestSolrCoreProperties, TestLeaderInitiatedRecoveryThread, TestClassicSimilarityFactory, TestManagedSchema, AssignTest, DateRangeFieldTest, TestConfigSetProperties, TestReRankQParserPlugin, TriLevelCompositeIdRoutingTest, TestSolrQueryParserResource, DistributedSuggestComponentTest, DocValuesTest, TestPartialUpdateDeduplication, TestCodecSupport, UpdateRequestProcessorFactoryTest, HdfsCollectionsAPIDistributedZkTest, LoggingHandlerTest, HdfsLockFactoryTest, TestQueryWrapperFilter, OpenExchangeRatesOrgProviderTest, HdfsRecoveryZkTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestLMJelinekMercerSimilarityFactory, TestSha256AuthenticationProvider, SolrCmdDistributorTest, TestSolr4Spatial2, OpenCloseCoreStressTest]
   [junit4] Completed [137/571 (1!)] on J1 in 689.93s, 6 tests, 1 error, 1 skipped <<< FAILURES!

[...truncated 1359 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/temp/junit4-J1-20151226_123647_584.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/heapdumps/java_pid8059.hprof ...
   [junit4] Heap dump file created [443868679 bytes in 6.146 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 16 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:791: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:728: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:991: There were test failures: 571 suites (8 ignored), 2276 tests, 1 error, 33 ignored (18 assumptions) [seed: F2D6F15CCEE2DB10]

Total time: 206 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 897 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/897/

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [TransactionLog]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [TransactionLog]
	at __randomizedtesting.SeedInfo.seed([D77AACF2B2B5CCE7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:229)
	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
	at __randomizedtesting.SeedInfo.seed([D77AACF2B2B5CCE7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:229)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory]
	at __randomizedtesting.SeedInfo.seed([D77AACF2B2B5CCE7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:229)
	at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10412 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/init-core-data-001
   [junit4]   2> 1341384 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[D77AACF2B2B5CCE7]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1341384 INFO  (SUITE-ChaosMonkeySafeLeaderTest-seed#[D77AACF2B2B5CCE7]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /a_y/
   [junit4]   2> 1341387 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1341387 INFO  (Thread-5363) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1341387 INFO  (Thread-5363) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1341487 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkTestServer start zk server on port:54397
   [junit4]   2> 1341487 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341488 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341490 INFO  (zkCallback-1523-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@298c9871 name:ZooKeeperConnection Watcher:127.0.0.1:54397 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341490 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341490 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341490 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1341492 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341494 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341496 INFO  (zkCallback-1524-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@40d20658 name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341496 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341496 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341496 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1341500 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1341503 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1341505 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1341508 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1341508 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1341512 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1341512 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1341514 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1341514 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1341516 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1341516 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1341518 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1341518 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1341520 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1341520 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1341525 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1341525 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1341527 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1341527 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1341530 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1341530 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1341533 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1341533 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1341537 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1341537 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1341541 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341542 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341543 INFO  (zkCallback-1525-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4544abca name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341543 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341544 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341872 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1
   [junit4]   2> 1341874 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 1341876 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@18f912e1{/a_y,null,AVAILABLE}
   [junit4]   2> 1341877 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.u.s.SslContextFactory x509=X509@2811708d(solrtest,h=[],w=[]) for SslContextFactory@24bac6db(file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore)
   [junit4]   2> 1341880 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.ServerConnector Started ServerConnector@780f7fe3{SSL,[ssl, http/1.1]}{127.0.0.1:41101}
   [junit4]   2> 1341880 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.Server Started @1344907ms
   [junit4]   2> 1341880 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/tempDir-001/control/data, hostContext=/a_y, hostPort=41101, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores}
   [junit4]   2> 1341880 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1341881 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001'
   [junit4]   2> 1341881 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1341881 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1341881 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341883 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341898 INFO  (zkCallback-1526-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2aecd07c name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341898 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341898 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341899 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1341899 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/solr.xml
   [junit4]   2> 1341909 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores
   [junit4]   2> 1341909 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer New CoreContainer 926607776
   [junit4]   2> 1341909 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001]
   [junit4]   2> 1341910 WARN  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/lib
   [junit4]   2> 1341911 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1341919 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1341921 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1341921 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1341922 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54397/solr
   [junit4]   2> 1341922 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1341922 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1341923 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341926 INFO  (zkCallback-1529-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@10eca9e9 name:ZooKeeperConnection Watcher:127.0.0.1:54397 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341927 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341931 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1341936 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1341941 INFO  (zkCallback-1530-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7df0de35 name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1341941 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1341943 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1341945 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1341949 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1341952 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1341955 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1341960 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1341961 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1341966 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1341967 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1341969 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41101_a_y
   [junit4]   2> 1341970 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:41101_a_y
   [junit4]   2> 1341972 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1341973 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1341975 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1341975 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95123646694686725-127.0.0.1:41101_a_y-n_0000000000
   [junit4]   2> 1341976 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41101_a_y
   [junit4]   2> 1341976 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1341977 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer Overseer (id=95123646694686725-127.0.0.1:41101_a_y-n_0000000000) starting
   [junit4]   2> 1341980 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1341991 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1341992 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1341993 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1341996 INFO  (OverseerCollectionConfigSetProcessor-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1341999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1342032 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1342032 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1342032 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores
   [junit4]   2> 1342033 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1342033 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1
   [junit4]   2> 1342033 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1342035 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1342035 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:41101_a_y    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1342036 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1342036 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1342037 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:41101/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:41101_a_y",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1342038 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1342038 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:41101/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:41101_a_y",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1342039 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1342039 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1342042 INFO  (zkCallback-1530-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1342043 INFO  (zkCallback-1530-thread-2-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1343038 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1343038 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1343039 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1343039 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1343040 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1343040 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1'
   [junit4]   2> 1343040 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1343040 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1343041 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1343047 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1343051 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1343062 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1343069 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1343182 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1343183 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1343184 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1343187 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1343210 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1343210 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1343210 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore solr.StandardDirectoryFactory
   [junit4]   2> 1343210 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/]
   [junit4]   2> 1343210 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@30f1226b
   [junit4]   2> 1343211 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data
   [junit4]   2> 1343211 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1343212 WARN  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1343212 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/index
   [junit4]   2> 1343212 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=69.69921875, floorSegmentMB=2.1015625, forceMergeDeletesPctAllowed=29.51221115478454, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1343222 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1343222 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1343223 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1343227 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1343228 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1343229 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1343229 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1343229 WARN  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1343232 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1343233 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1343233 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1343234 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1343237 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1343238 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1343238 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1343238 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1343240 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 1343240 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1343240 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=41, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1343241 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 1343241 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1343241 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@7698919f[collection1] main
   [junit4]   2> 1343241 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1343245 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1343245 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1343248 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1343248 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1343248 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1343248 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1343248 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1343249 INFO  (searcherExecutor-2511-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7698919f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1343249 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1521978349065338880
   [junit4]   2> 1343252 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 3.0ms to seed version buckets with highest version 1521978349065338880
   [junit4]   2> 1343252 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1343253 INFO  (coreLoadExecutor-2510-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1343254 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:https://127.0.0.1:41101/a_y collection:control_collection shard:shard1
   [junit4]   2> 1343255 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1343262 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95123646694686725-core_node1-n_0000000000
   [junit4]   2> 1343262 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 1343264 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1343264 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1343264 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1343264 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:41101/a_y/collection1/
   [junit4]   2> 1343264 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1343264 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:41101/a_y/collection1/ has no replicas
   [junit4]   2> 1343266 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1343271 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 1343274 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:41101/a_y/collection1/ shard1
   [junit4]   2> 1343274 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"https://127.0.0.1:41101/a_y",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1343378 INFO  (zkCallback-1530-thread-2-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1343378 INFO  (zkCallback-1530-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1343427 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are https://127.0.0.1:41101/a_y/collection1/ and leader is https://127.0.0.1:41101/a_y/collection1/
   [junit4]   2> 1343427 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:41101/a_y
   [junit4]   2> 1343427 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1343427 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1343427 INFO  (coreZkRegister-2504-thread-1-processing-n:127.0.0.1:41101_a_y x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41101_a_y c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1343429 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:41101/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:41101_a_y",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 1343429 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:41101/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:41101_a_y",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1343532 INFO  (zkCallback-1530-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1343532 INFO  (zkCallback-1530-thread-2-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1343540 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1343559 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1343571 INFO  (zkCallback-1532-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6c0cd8cd name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1343572 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1343572 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1343572 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1343574 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1343574 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1343574 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1343587 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1343596 INFO  (zkCallback-1533-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@54954eb4 name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1343596 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1343596 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1343597 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"4",
   [junit4]   2>   "stateFormat":"2"} current state version: 3
   [junit4]   2> 1343598 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 1343700 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 1343953 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores/collection1
   [junit4]   2> 1343954 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001
   [junit4]   2> 1343955 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 1343957 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@58785e2a{/a_y,null,AVAILABLE}
   [junit4]   2> 1343958 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.u.s.SslContextFactory x509=X509@25c57cf0(solrtest,h=[],w=[]) for SslContextFactory@1a05ce2c(file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore)
   [junit4]   2> 1343977 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.ServerConnector Started ServerConnector@760b2cd8{SSL,[ssl, http/1.1]}{127.0.0.1:34831}
   [junit4]   2> 1343980 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.e.j.s.Server Started @1347007ms
   [junit4]   2> 1343980 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/a_y, hostPort=34831, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores}
   [junit4]   2> 1343980 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1343981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001'
   [junit4]   2> 1343981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1343981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1343981 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1343983 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1343985 INFO  (zkCallback-1534-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@43114d97 name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1343985 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1343985 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1343986 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1343986 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/solr.xml
   [junit4]   2> 1343991 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores
   [junit4]   2> 1343991 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer New CoreContainer 2085268738
   [junit4]   2> 1343991 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001]
   [junit4]   2> 1343991 WARN  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/lib
   [junit4]   2> 1343995 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1343997 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54397/solr
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1343999 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1344001 INFO  (zkCallback-1537-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@344336b5 name:ZooKeeperConnection Watcher:127.0.0.1:54397 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1344002 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1344002 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1344004 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1344005 INFO  (zkCallback-1538-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7bc93874 name:ZooKeeperConnection Watcher:127.0.0.1:54397/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1344005 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1344012 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1345018 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34831_a_y
   [junit4]   2> 1345019 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:34831_a_y
   [junit4]   2> 1345020 INFO  (zkCallback-1532-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1345020 INFO  (zkCallback-1538-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1345020 INFO  (zkCallback-1530-thread-2-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1345020 INFO  (zkCallback-1530-thread-1-processing-n:127.0.0.1:41101_a_y) [n:127.0.0.1:41101_a_y    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1345022 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1345022 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95123646694686730-127.0.0.1:34831_a_y-n_0000000001
   [junit4]   2> 1345023 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95123646694686725-127.0.0.1:41101_a_y-n_0000000000 to know if I could be the leader
   [junit4]   2> 1345033 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1345033 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1345034 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores
   [junit4]   2> 1345036 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1345036 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores/collection1
   [junit4]   2> 1345036 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1345054 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1345054 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1345055 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:34831/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:34831_a_y",
   [junit4]   2>   "numShards":"4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 1345055 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController Registering watch for collection collection1
   [junit4]   2> 1345056 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [collection1]
   [junit4]   2> 1345056 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Add data for [collection1] ver [0]
   [junit4]   2> 1345056 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1345056 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Update state numShards=4 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:34831/a_y",
   [junit4]   2>   "node_name":"127.0.0.1:34831_a_y",
   [junit4]   2>   "numShards":"4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1345056 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=4
   [junit4]   2> 1345056 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1345058 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1345058 INFO  (TEST-ChaosMonkeySafeLeaderTest.test-seed#[D77AACF2B2B5CCE7]) [n:127.0.0.1:34831_a_y    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1345168 INFO  (OverseerStateUpdate-95123646694686725-127.0.0.1:41101_a_y-n_0000000000) [n:127.0.0.1:41101_a_y    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 1345168 INFO  (zkCallback-1538-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1345169 INFO  (zkCallback-1538-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y    ] o.a.s.c.c.ZkStateReader Updating data for [collection1] from [0] to [1]
   [junit4]   2> 1346058 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1346058 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController Check for collection zkNode:collection1
   [junit4]   2> 1346059 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1346059 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/collection1]
   [junit4]   2> 1346059 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/collection1] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1346059 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest_D77AACF2B2B5CCE7-001/shard-1-001/cores/collection1'
   [junit4]   2> 1346059 INFO  (coreLoadExecutor-2521-thread-1-processing-n:127.0.0.1:34831_a_y) [n:127.0.0.1:34831_a_y c:collection1   x:collection1

[...truncated too long message...]

lr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:226)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   [junit4]   2> 	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:45)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:517)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> Dec 30, 2015 11:12:55 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {rnd_b=BlockTreeOrds(blocksize=128), _version_=PostingsFormat(name=Memory doPackFST= false), a_t=PostingsFormat(name=Memory doPackFST= false), a_i=BlockTreeOrds(blocksize=128), id=BlockTreeOrds(blocksize=128)}, docValues:{}, sim=ClassicSimilarity, locale=es_MX, timezone=Asia/Beirut
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=4,threads=2,free=314799256,total=515899392
   [junit4]   2> NOTE: All tests run in this JVM: [ForceLeaderTest, FullHLLTest, TestUtils, TestSweetSpotSimilarityFactory, TestDocBasedVersionConstraints, AnalysisErrorHandlingTest, TestRTimerTree, HardAutoCommitTest, TestJmxIntegration, UUIDUpdateProcessorFallbackTest, BlockJoinFacetSimpleTest, TestSolrIndexConfig, CloneFieldUpdateProcessorFactoryTest, TestDefaultStatsCache, TestShortCircuitedRequests, TestScoreJoinQPScore, AnalyticsMergeStrategyTest, TestCSVResponseWriter, DebugComponentTest, OverseerTest, TestComplexPhraseQParserPlugin, TestIndexSearcher, CoreAdminRequestStatusTest, HdfsChaosMonkeyNothingIsSafeTest, TestXIncludeConfig, XmlUpdateRequestHandlerTest, PrimitiveFieldTypeTest, DocValuesMultiTest, TestConfigSetsAPI, SuggestComponentTest, SolrTestCaseJ4Test, HdfsRestartWhileUpdatingTest, TestStressUserVersions, PluginInfoTest, TestInfoStreamLogging, ExitableDirectoryReaderTest, BadIndexSchemaTest, TestMiniSolrCloudClusterSSL, DirectUpdateHandlerTest, OverseerCollectionConfigSetProcessorTest, SmileWriterTest, TestExtendedDismaxParser, TestFieldTypeResource, CollectionTooManyReplicasTest, TestConfigSets, ScriptEngineTest, TestSolrJ, LukeRequestHandlerTest, BadCopyFieldTest, MergeStrategyTest, TestReloadDeadlock, TestCollationField, TestBulkSchemaConcurrent, TestRuleBasedAuthorizationPlugin, QueryParsingTest, TestBulkSchemaAPI, TestBlobHandler, DistributedExpandComponentTest, RequestHandlersTest, NoCacheHeaderTest, SolrCloudExampleTest, TestRawTransformer, VMParamsZkACLAndCredentialsProvidersTest, TestReqParamsAPI, TestUniqueKeyFieldResource, FieldMutatingUpdateProcessorTest, UnloadDistributedZkTest, TestClusterStateMutator, TestFieldCollectionResource, DistributedFacetPivotSmallAdvancedTest, SecurityConfHandlerTest, TestPHPSerializedResponseWriter, TestGroupingSearch, TestMaxScoreQueryParser, BlockCacheTest, BasicDistributedZk2Test, SolrCoreCheckLockOnStartupTest, ParsingFieldUpdateProcessorsTest, TestSchemaManager, DistribJoinFromCollectionTest, TestSerializedLuceneMatchVersion, TestExactStatsCache, DocValuesMissingTest, TestWriterPerf, TestOverriddenPrefixQueryForCustomFieldType, TestSolrQueryParser, DistributedQueryComponentCustomSortTest, BadComponentTest, TestDefaultSearchFieldResource, CircularListTest, DirectUpdateHandlerOptimizeTest, HighlighterMaxOffsetTest, TestFilteredDocIdSet, TestStressRecovery, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, LeaderElectionIntegrationTest, TestReplicationHandler, DistributedTermsComponentTest, TestSolr4Spatial, StatsComponentTest, BasicFunctionalityTest, TestLazyCores, TestBadConfig, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SuggesterFSTTest, CoreAdminHandlerTest, TestStressLucene, SpatialFilterTest, WordBreakSolrSpellCheckerTest, TestWordDelimiterFilterFactory, TestSolrDeletionPolicy1, SolrInfoMBeanTest, CacheHeaderTest, TestQueryUtils, TestReversedWildcardFilterFactory, TestOmitPositions, TermsComponentTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, MoreLikeThisHandlerTest, TestArbitraryIndexDir, RegexBoostProcessorTest, SolrPluginUtilsTest, UniqFieldsUpdateProcessorFactoryTest, PingRequestHandlerTest, DirectSolrConnectionTest, SpellPossibilityIteratorTest, TestCharFilters, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, TestJmxMonitoredMap, URLClassifyProcessorTest, DateMathParserTest, TestSystemIdResolver, TestSuggestSpellingConverter, SpellingQueryConverterTest, DOMUtilTest, ClusterStateTest, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, DistributedIntervalFacetingTest, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestTolerantSearch, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, ActionThrottleTest, AssignTest, AsyncCallRequestStatusResponseTest, CdcrReplicationDistributedZkTest, ReplicationFactorTest, SSLMigrationTest, SaslZkACLProviderTest, SimpleCollectionCreateDeleteTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudInspectUtil, TestCloudPivotFacet, TestDistribDocBasedVersion, TestMiniSolrCloudCluster, HdfsChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsChaosMonkeySafeLeaderTest -Dtests.seed=D77AACF2B2B5CCE7 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_MX -Dtests.timezone=Asia/Beirut -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | HdfsChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D77AACF2B2B5CCE7]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:229)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [483/573 (3!)] on J2 in 118.03s, 1 test, 1 failure <<< FAILURES!

[...truncated 301 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:791: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:728: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1424: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:981: There were test failures: 573 suites (8 ignored), 2284 tests, 3 suite-level errors, 33 ignored (18 assumptions) [seed: D77AACF2B2B5CCE7]

Total time: 205 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Re: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 896 - Still Failing

Posted by Michael McCandless <lu...@mikemccandless.com>.
I'll dig, silly test bug!

Mike McCandless

http://blog.mikemccandless.com


On Tue, Dec 29, 2015 at 4:11 AM, Apache Jenkins Server
<je...@builds.apache.org> wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/896/
>
> 1 tests failed.
> FAILED:  org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny
>
> Error Message:
> maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode
>
> Stack Trace:
> java.lang.IllegalArgumentException: maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode
>         at __randomizedtesting.SeedInfo.seed([B6A7B02C094A1978:181EDD6FE19860C1]:0)
>         at org.apache.lucene.util.bkd.BKDWriter.<init>(BKDWriter.java:161)
>         at org.apache.lucene.codecs.lucene60.Lucene60DimensionalWriter.writeField(Lucene60DimensionalWriter.java:88)
>         at org.apache.lucene.index.DimensionalValuesWriter.flush(DimensionalValuesWriter.java:68)
>         at org.apache.lucene.index.DefaultIndexingChain.writeDimensionalValues(DefaultIndexingChain.java:146)
>         at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:96)
>         at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:425)
>         at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
>         at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
>         at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3099)
>         at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3074)
>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1727)
>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1707)
>         at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:421)
>         at org.apache.lucene.search.TestDimensionalRangeQuery.verifyBinary(TestDimensionalRangeQuery.java:508)
>         at org.apache.lucene.search.TestDimensionalRangeQuery.doTestRandomBinary(TestDimensionalRangeQuery.java:419)
>         at org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny(TestDimensionalRangeQuery.java:375)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:497)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> Build Log:
> [...truncated 1700 lines...]
>    [junit4] Suite: org.apache.lucene.search.TestDimensionalRangeQuery
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDimensionalRangeQuery -Dtests.method=testRandomBinaryTiny -Dtests.seed=B6A7B02C094A1978 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_BO -Dtests.timezone=SystemV/EST5 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>    [junit4] ERROR   0.02s J0 | TestDimensionalRangeQuery.testRandomBinaryTiny <<<
>    [junit4]    > Throwable #1: java.lang.IllegalArgumentException: maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([B6A7B02C094A1978:181EDD6FE19860C1]:0)
>    [junit4]    >        at org.apache.lucene.util.bkd.BKDWriter.<init>(BKDWriter.java:161)
>    [junit4]    >        at org.apache.lucene.codecs.lucene60.Lucene60DimensionalWriter.writeField(Lucene60DimensionalWriter.java:88)
>    [junit4]    >        at org.apache.lucene.index.DimensionalValuesWriter.flush(DimensionalValuesWriter.java:68)
>    [junit4]    >        at org.apache.lucene.index.DefaultIndexingChain.writeDimensionalValues(DefaultIndexingChain.java:146)
>    [junit4]    >        at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:96)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:425)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
>    [junit4]    >        at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3099)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3074)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1727)
>    [junit4]    >        at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1707)
>    [junit4]    >        at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:421)
>    [junit4]    >        at org.apache.lucene.search.TestDimensionalRangeQuery.verifyBinary(TestDimensionalRangeQuery.java:508)
>    [junit4]    >        at org.apache.lucene.search.TestDimensionalRangeQuery.doTestRandomBinary(TestDimensionalRangeQuery.java:419)
>    [junit4]    >        at org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny(TestDimensionalRangeQuery.java:375)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/core/test/J0/temp/lucene.search.TestDimensionalRangeQuery_B6A7B02C094A1978-001
>    [junit4]   2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity, locale=es_BO, timezone=SystemV/EST5
>    [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=4,threads=1,free=187269408,total=429391872
>    [junit4]   2> NOTE: All tests run in this JVM: [TestSegmentInfos, TestConjunctionDISI, TestLucene54DocValuesFormat, TestMultiDocValues, TestBlockPostingsFormat3, TestPagedBytes, TestSentinelIntSet, TestNumericRangeQuery64, TestTermScorer, TestNativeFSLockFactory, TestLucene50FieldInfoFormat, TestVirtualMethod, TestFixedBitSet, TestIndexWriterUnicode, TestRollingBuffer, TestInfoStream, TestOrdinalMap, TestTerms, TestDocInverterPerFieldErrorInfo, Test2BNumericDocValues, TestGrowableByteArrayDataOutput, TestMergedIterator, FuzzyTermOnShortTermsTest, TestSegmentReader, TestAttributeSource, TestSloppyMath, TestWeakIdentityMap, TestDeterminism, TestIndexWriterExceptions2, TestTimeLimitingCollector, TestIndexSearcher, TestByteBlockPool, TestParallelTermEnum, TestIndexWriterCommit, TestDoc, TestCustomNorms, TestMinimize, TestIndexWriterOnVMError, TestBooleanMinShouldMatch, TestUniqueTermCount, TestFilterDirectory, TestOperations, TestStringHelper, FiniteStringsIteratorTest, TestNoMergePolicy, TestIndexWriterMergePolicy, TestMaxPosition, TestSizeBoundedForceMerge, TestSpanBoostQuery, TestAutomatonQuery, TestSimilarity2, TestSloppyPhraseQuery2, TestParallelCompositeReader, TestPerFieldPostingsFormat, MultiCollectorTest, Test2BSortedDocValuesOrds, TestUnicodeUtil, TestDocValuesIndexing, TestIndexWriterLockRelease, TestMultiTermConstantScore, TestThreadedForceMerge, TestForTooMuchCloning, TestPositionIncrement, TestMutableValues, TestReusableStringReader, TestIndexWriterConfig, TestSearcherManager, TestHighCompressionMode, TestDocumentsWriterStallControl, TestExceedMaxTermLength, TestFilterDirectoryReader, TestIndexReaderClose, TestIndexWriterFromReader, TestIndexWriterThreadsToSegments, TestReaderWrapperDVTypeCheck, TestStressDeletes, TestApproximationSearchEquivalence, TestBooleanCoord, TestBooleanRewrites, TestControlledRealTimeReopenThread, TestDimensionalRangeQuery]
>    [junit4] Completed [403/403 (1!)] on J0 in 757.26s, 18 tests, 1 error <<< FAILURES!
>
> [...truncated 11 lines...]
> BUILD FAILED
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:791: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:728: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:50: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1436: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:991: There were test failures: 403 suites (1 ignored), 3317 tests, 1 error, 29 ignored (25 assumptions) [seed: B6A7B02C094A1978]
>
> Total time: 61 minutes 45 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> No prior successful build to compare, so performing full copy of artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 896 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/896/

1 tests failed.
FAILED:  org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny

Error Message:
maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode

Stack Trace:
java.lang.IllegalArgumentException: maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode
	at __randomizedtesting.SeedInfo.seed([B6A7B02C094A1978:181EDD6FE19860C1]:0)
	at org.apache.lucene.util.bkd.BKDWriter.<init>(BKDWriter.java:161)
	at org.apache.lucene.codecs.lucene60.Lucene60DimensionalWriter.writeField(Lucene60DimensionalWriter.java:88)
	at org.apache.lucene.index.DimensionalValuesWriter.flush(DimensionalValuesWriter.java:68)
	at org.apache.lucene.index.DefaultIndexingChain.writeDimensionalValues(DefaultIndexingChain.java:146)
	at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:96)
	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:425)
	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3099)
	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3074)
	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1727)
	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1707)
	at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:421)
	at org.apache.lucene.search.TestDimensionalRangeQuery.verifyBinary(TestDimensionalRangeQuery.java:508)
	at org.apache.lucene.search.TestDimensionalRangeQuery.doTestRandomBinary(TestDimensionalRangeQuery.java:419)
	at org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny(TestDimensionalRangeQuery.java:375)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 1700 lines...]
   [junit4] Suite: org.apache.lucene.search.TestDimensionalRangeQuery
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDimensionalRangeQuery -Dtests.method=testRandomBinaryTiny -Dtests.seed=B6A7B02C094A1978 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es_BO -Dtests.timezone=SystemV/EST5 -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.02s J0 | TestDimensionalRangeQuery.testRandomBinaryTiny <<<
   [junit4]    > Throwable #1: java.lang.IllegalArgumentException: maxMBSortInHeap=2.255092363313635 only allows for maxPointsSortInHeap=1443, but this is less than maxPointsInLeafNode=1606; either increase maxMBSortInHeap or decrease maxPointsInLeafNode
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B6A7B02C094A1978:181EDD6FE19860C1]:0)
   [junit4]    > 	at org.apache.lucene.util.bkd.BKDWriter.<init>(BKDWriter.java:161)
   [junit4]    > 	at org.apache.lucene.codecs.lucene60.Lucene60DimensionalWriter.writeField(Lucene60DimensionalWriter.java:88)
   [junit4]    > 	at org.apache.lucene.index.DimensionalValuesWriter.flush(DimensionalValuesWriter.java:68)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.writeDimensionalValues(DefaultIndexingChain.java:146)
   [junit4]    > 	at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:96)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:425)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)
   [junit4]    > 	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3099)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3074)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1727)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1707)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:421)
   [junit4]    > 	at org.apache.lucene.search.TestDimensionalRangeQuery.verifyBinary(TestDimensionalRangeQuery.java:508)
   [junit4]    > 	at org.apache.lucene.search.TestDimensionalRangeQuery.doTestRandomBinary(TestDimensionalRangeQuery.java:419)
   [junit4]    > 	at org.apache.lucene.search.TestDimensionalRangeQuery.testRandomBinaryTiny(TestDimensionalRangeQuery.java:375)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build/core/test/J0/temp/lucene.search.TestDimensionalRangeQuery_B6A7B02C094A1978-001
   [junit4]   2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity, locale=es_BO, timezone=SystemV/EST5
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=4,threads=1,free=187269408,total=429391872
   [junit4]   2> NOTE: All tests run in this JVM: [TestSegmentInfos, TestConjunctionDISI, TestLucene54DocValuesFormat, TestMultiDocValues, TestBlockPostingsFormat3, TestPagedBytes, TestSentinelIntSet, TestNumericRangeQuery64, TestTermScorer, TestNativeFSLockFactory, TestLucene50FieldInfoFormat, TestVirtualMethod, TestFixedBitSet, TestIndexWriterUnicode, TestRollingBuffer, TestInfoStream, TestOrdinalMap, TestTerms, TestDocInverterPerFieldErrorInfo, Test2BNumericDocValues, TestGrowableByteArrayDataOutput, TestMergedIterator, FuzzyTermOnShortTermsTest, TestSegmentReader, TestAttributeSource, TestSloppyMath, TestWeakIdentityMap, TestDeterminism, TestIndexWriterExceptions2, TestTimeLimitingCollector, TestIndexSearcher, TestByteBlockPool, TestParallelTermEnum, TestIndexWriterCommit, TestDoc, TestCustomNorms, TestMinimize, TestIndexWriterOnVMError, TestBooleanMinShouldMatch, TestUniqueTermCount, TestFilterDirectory, TestOperations, TestStringHelper, FiniteStringsIteratorTest, TestNoMergePolicy, TestIndexWriterMergePolicy, TestMaxPosition, TestSizeBoundedForceMerge, TestSpanBoostQuery, TestAutomatonQuery, TestSimilarity2, TestSloppyPhraseQuery2, TestParallelCompositeReader, TestPerFieldPostingsFormat, MultiCollectorTest, Test2BSortedDocValuesOrds, TestUnicodeUtil, TestDocValuesIndexing, TestIndexWriterLockRelease, TestMultiTermConstantScore, TestThreadedForceMerge, TestForTooMuchCloning, TestPositionIncrement, TestMutableValues, TestReusableStringReader, TestIndexWriterConfig, TestSearcherManager, TestHighCompressionMode, TestDocumentsWriterStallControl, TestExceedMaxTermLength, TestFilterDirectoryReader, TestIndexReaderClose, TestIndexWriterFromReader, TestIndexWriterThreadsToSegments, TestReaderWrapperDVTypeCheck, TestStressDeletes, TestApproximationSearchEquivalence, TestBooleanCoord, TestBooleanRewrites, TestControlledRealTimeReopenThread, TestDimensionalRangeQuery]
   [junit4] Completed [403/403 (1!)] on J0 in 757.26s, 18 tests, 1 error <<< FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:791: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:728: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/build.xml:50: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:991: There were test failures: 403 suites (1 ignored), 3317 tests, 1 error, 29 ignored (25 assumptions) [seed: B6A7B02C094A1978]

Total time: 61 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 895 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/895/

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=22616, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=22616, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:46930, https://127.0.0.1:59010, https://127.0.0.1:42959, https://127.0.0.1:40538, https://127.0.0.1:56851]
	at __randomizedtesting.SeedInfo.seed([8AF97C780B82D319]:0)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:891)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:46930, https://127.0.0.1:59010, https://127.0.0.1:42959, https://127.0.0.1:40538, https://127.0.0.1:56851]
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:871)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1574)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:881)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:42959: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qnr-
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
	... 7 more




Build Log:
[...truncated 10867 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/init-core-data-001
   [junit4]   2> 1557779 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[8AF97C780B82D319]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1557779 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[8AF97C780B82D319]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1557784 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1557784 INFO  (Thread-13787) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1557784 INFO  (Thread-13787) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1557885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.ZkTestServer start zk server on port:54894
   [junit4]   2> 1557885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1557885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1557888 INFO  (zkCallback-2464-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@488a36d9 name:ZooKeeperConnection Watcher:127.0.0.1:54894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1557888 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1557888 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1557888 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1557895 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1557895 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1557896 INFO  (zkCallback-2465-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7446a2cf name:ZooKeeperConnection Watcher:127.0.0.1:54894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1557896 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1557896 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1557896 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1557898 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1557899 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1557903 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1557904 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1557904 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1557906 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1557906 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1557908 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1557908 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1557910 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1557910 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1557911 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1557911 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1557913 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1557913 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1557914 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1557915 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1557916 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1557916 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1557918 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1557918 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1557919 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1557919 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1557921 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1557921 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1557923 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1557923 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1557924 INFO  (zkCallback-2466-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@50c2130c name:ZooKeeperConnection Watcher:127.0.0.1:54894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1557924 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1557924 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1558249 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1
   [junit4]   2> 1558252 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 1558253 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@24725791{/,null,AVAILABLE}
   [junit4]   2> 1558254 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.e.j.u.s.SslContextFactory x509=X509@287a6f2c(solrtest,h=[],w=[]) for SslContextFactory@40dd7394(file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore)
   [junit4]   2> 1558256 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.e.j.s.ServerConnector Started ServerConnector@74f6f09b{SSL,[ssl, http/1.1]}{127.0.0.1:42959}
   [junit4]   2> 1558256 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.e.j.s.Server Started @1561530ms
   [junit4]   2> 1558257 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/tempDir-001/control/data, hostContext=/, hostPort=42959, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores}
   [junit4]   2> 1558258 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1558258 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001'
   [junit4]   2> 1558258 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1558258 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1558258 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1558259 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1558259 INFO  (zkCallback-2467-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@163b032 name:ZooKeeperConnection Watcher:127.0.0.1:54894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1558260 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1558260 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1558261 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1558261 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/solr.xml
   [junit4]   2> 1558265 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores
   [junit4]   2> 1558266 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.CoreContainer New CoreContainer 1350153798
   [junit4]   2> 1558266 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001]
   [junit4]   2> 1558266 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/lib
   [junit4]   2> 1558266 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1558268 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1558269 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1558269 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1558269 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54894/solr
   [junit4]   2> 1558269 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1558269 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1558270 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1558271 INFO  (zkCallback-2470-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4c7ba905 name:ZooKeeperConnection Watcher:127.0.0.1:54894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1558271 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1558272 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1558273 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1558274 INFO  (zkCallback-2471-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@22e58e02 name:ZooKeeperConnection Watcher:127.0.0.1:54894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1558274 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1558275 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1558277 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1558279 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1558281 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1558283 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1558285 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1558286 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1558287 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1558290 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1558291 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42959_
   [junit4]   2> 1558291 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:42959_
   [junit4]   2> 1558293 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1558295 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1558297 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1558297 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95112366953725957-127.0.0.1:42959_-n_0000000000
   [junit4]   2> 1558297 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42959_
   [junit4]   2> 1558297 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1558299 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.Overseer Overseer (id=95112366953725957-127.0.0.1:42959_-n_0000000000) starting
   [junit4]   2> 1558300 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1558305 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1558305 INFO  (OverseerCollectionConfigSetProcessor-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1558306 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1558306 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1558332 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1558332 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1558332 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores
   [junit4]   2> 1558333 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1558334 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1
   [junit4]   2> 1558334 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1558335 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 1558335 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:42959_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1558335 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1558335 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1558336 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:42959",
   [junit4]   2>   "node_name":"127.0.0.1:42959_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1558337 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1558338 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:42959",
   [junit4]   2>   "node_name":"127.0.0.1:42959_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1558338 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1558338 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1558340 INFO  (zkCallback-2471-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1559338 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1559338 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1559339 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1559339 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1559339 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1559339 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1'
   [junit4]   2> 1559340 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1559340 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1559341 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1559346 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1559349 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1559357 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1559362 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1559449 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1559454 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1559455 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1559462 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1559464 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1559465 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1559466 WARN  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1559466 WARN  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1559466 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1559467 WARN  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1559467 WARN  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1559468 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 1559468 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1559468 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1559468 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1/data/]
   [junit4]   2> 1559468 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@38ba9796
   [junit4]   2> 1559469 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1/data
   [junit4]   2> 1559469 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1559469 WARN  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1559470 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1/data/index
   [junit4]   2> 1559470 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=3, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5015402840642721]
   [junit4]   2> 1559470 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@5819001c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@52fb86c0),segFN=segments_1,generation=1}
   [junit4]   2> 1559470 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1559471 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1/data/
   [junit4]   2> 1559471 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1559474 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1559474 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1559474 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1559474 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1559474 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1559474 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1559475 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1559475 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1559475 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1559475 WARN  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1559476 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1559477 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1559478 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1559479 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1559482 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1559482 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1559483 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1559483 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1559484 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1559484 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1559484 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=2, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1559484 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@5819001c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@52fb86c0),segFN=segments_1,generation=1}
   [junit4]   2> 1559484 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1559484 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@106834df[collection1] main
   [junit4]   2> 1559485 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1559485 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1559485 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1559486 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1559486 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1559486 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1559486 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1559486 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1559486 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1559487 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1559487 INFO  (searcherExecutor-6003-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@106834df[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1559487 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1559487 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1559487 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1559487 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1521797873044291584
   [junit4]   2> 1559489 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 1521797873044291584
   [junit4]   2> 1559489 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1559490 INFO  (coreLoadExecutor-6002-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1559490 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:https://127.0.0.1:42959 collection:control_collection shard:shard1
   [junit4]   2> 1559491 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1559494 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95112366953725957-core_node1-n_0000000000
   [junit4]   2> 1559494 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 1559496 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1559496 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1559496 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1559496 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:42959/collection1/
   [junit4]   2> 1559496 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1559496 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:42959/collection1/ has no replicas
   [junit4]   2> 1559497 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1559499 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 1559501 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:42959/collection1/ shard1
   [junit4]   2> 1559501 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"https://127.0.0.1:42959",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1559603 INFO  (zkCallback-2471-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1559652 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are https://127.0.0.1:42959/collection1/ and leader is https://127.0.0.1:42959/collection1/
   [junit4]   2> 1559652 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:42959
   [junit4]   2> 1559652 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1559652 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1559652 INFO  (coreZkRegister-5996-thread-1-processing-n:127.0.0.1:42959_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:42959_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1559654 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:42959",
   [junit4]   2>   "node_name":"127.0.0.1:42959_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 1559654 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:42959",
   [junit4]   2>   "node_name":"127.0.0.1:42959_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1559757 INFO  (zkCallback-2471-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1559838 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1559839 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1559840 INFO  (zkCallback-2473-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3c0c4be2 name:ZooKeeperConnection Watcher:127.0.0.1:54894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1559840 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1559840 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1559840 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1559842 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1559842 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1559842 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1559846 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1559846 INFO  (zkCallback-2474-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2565db5e name:ZooKeeperConnection Watcher:127.0.0.1:54894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1559846 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1559846 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1559850 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 3
   [junit4]   2> 1559850 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 1559952 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 1560173 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001/cores/collection1
   [junit4]   2> 1560173 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001
   [junit4]   2> 1560175 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 1560176 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@22bd1a5d{/,null,AVAILABLE}
   [junit4]   2> 1560177 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.e.j.u.s.SslContextFactory x509=X509@68e63a93(solrtest,h=[],w=[]) for SslContextFactory@2b242602(file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/server/etc/test/solrtest.keystore)
   [junit4]   2> 1560178 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.e.j.s.ServerConnector Started ServerConnector@2da5611c{SSL,[ssl, http/1.1]}{127.0.0.1:46930}
   [junit4]   2> 1560179 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.e.j.s.Server Started @1563452ms
   [junit4]   2> 1560179 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=46930, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001/cores}
   [junit4]   2> 1560180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1560180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001'
   [junit4]   2> 1560180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1560180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1560180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1560181 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1560182 INFO  (zkCallback-2475-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2b107078 name:ZooKeeperConnection Watcher:127.0.0.1:54894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1560182 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1560182 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1560183 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1560183 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001/solr.xml
   [junit4]   2> 1560187 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001/cores
   [junit4]   2> 1560188 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.CoreContainer New CoreContainer 1560035836
   [junit4]   2> 1560188 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001]
   [junit4]   2> 1560188 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001/lib
   [junit4]   2> 1560188 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1560190 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1560191 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1560191 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1560192 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54894/solr
   [junit4]   2> 1560192 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1560192 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1560192 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1560193 INFO  (zkCallback-2478-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6be596bd name:ZooKeeperConnection Watcher:127.0.0.1:54894 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1560194 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1560194 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1560196 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1560196 INFO  (zkCallback-2479-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@47ae1e05 name:ZooKeeperConnection Watcher:127.0.0.1:54894/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1560196 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1560201 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1561206 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46930_
   [junit4]   2> 1561207 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:46930_
   [junit4]   2> 1561208 INFO  (zkCallback-2473-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1561208 INFO  (zkCallback-2471-thread-1-processing-n:127.0.0.1:42959_) [n:127.0.0.1:42959_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1561208 INFO  (zkCallback-2479-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1561210 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1561210 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95112366953725962-127.0.0.1:46930_-n_0000000001
   [junit4]   2> 1561211 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95112366953725957-127.0.0.1:42959_-n_0000000000 to know if I could be the leader
   [junit4]   2> 1561222 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1561222 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1561223 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001/cores
   [junit4]   2> 1561223 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1561224 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001/cores/collection1
   [junit4]   2> 1561224 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1561224 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 1561224 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[8AF97C780B82D319]) [n:127.0.0.1:46930_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1561225 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1561225 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1561226 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.ZkController Registering watch for collection collection1
   [junit4]   2> 1561226 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [collection1]
   [junit4]   2> 1561227 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:46930",
   [junit4]   2>   "node_name":"127.0.0.1:46930_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 1561227 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Add data for [collection1] ver [0]
   [junit4]   2> 1561227 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1561227 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:46930",
   [junit4]   2>   "node_name":"127.0.0.1:46930_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1561227 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=2
   [junit4]   2> 1561227 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1561329 INFO  (OverseerStateUpdate-95112366953725957-127.0.0.1:42959_-n_0000000000) [n:127.0.0.1:42959_    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 1561330 INFO  (zkCallback-2479-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1561330 INFO  (zkCallback-2479-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_    ] o.a.s.c.c.ZkStateReader Updating data for [collection1] from [0] to [1]
   [junit4]   2> 1562227 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1562227 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.ZkController Check for collection zkNode:collection1
   [junit4]   2> 1562228 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1562228 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/collection1]
   [junit4]   2> 1562228 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/collection1] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1562228 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_8AF97C780B82D319-001/shard-1-001/cores/collection1'
   [junit4]   2> 1562228 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1562228 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1562230 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1562235 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1562239 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1562259 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1562264 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1562341 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1562356 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1562357 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1562364 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1562366 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1562368 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1562368 WARN  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1562369 WARN  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1562369 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1562369 WARN  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1562369 WARN  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1562370 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 1562370 INFO  (coreLoadExecutor-6013-thread-1-processing-n:127.0.0.1:46930_) [n:127.0.0.1:46930_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1562370 INF

[...truncated too long message...]

o.seed([8AF97C780B82D319]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:891)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:46930, https://127.0.0.1:59010, https://127.0.0.1:42959, https://127.0.0.1:40538, https://127.0.0.1:56851]
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:871)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1574)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:881)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:42959: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qnr-
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
   [junit4]    > 	... 7 moreThrowable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=22617, name=collection5, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:46930, https://127.0.0.1:59010, https://127.0.0.1:42959, https://127.0.0.1:40538, https://127.0.0.1:56851]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8AF97C780B82D319]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:891)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:46930, https://127.0.0.1:59010, https://127.0.0.1:42959, https://127.0.0.1:40538, https://127.0.0.1:56851]
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:871)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1574)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:881)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:42959: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qnr-
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
   [junit4]    > 	... 7 more
   [junit4]   2> 2158194 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[8AF97C780B82D319]-worker) [n:127.0.0.1:59010_ c:addReplicaColl s:shard1 r:core_node7 x:propertyDotName] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {other_tl1=PostingsFormat(name=Memory doPackFST= false), range_facet_l_dv=PostingsFormat(name=Direct), _version_=PostingsFormat(name=Memory doPackFST= false), multiDefault=PostingsFormat(name=LuceneFixedGap), a_t=PostingsFormat(name=Memory doPackFST= false), intDefault=PostingsFormat(name=Memory doPackFST= false), a_i1=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=Direct), range_facet_i_dv=PostingsFormat(name=Memory doPackFST= false), text=FSTOrd50, range_facet_l=PostingsFormat(name=Memory doPackFST= false), timestamp=PostingsFormat(name=Memory doPackFST= false)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene54), range_facet_i_dv=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Memory)}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=hu_HU, timezone=Atlantic/Faeroe
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=4,threads=1,free=74825376,total=498597888
   [junit4]   2> NOTE: All tests run in this JVM: [TestDistributedGrouping, TestCloudManagedSchemaConcurrent, ResponseHeaderTest, ShardRoutingTest, AtomicUpdatesTest, NumericFieldsTest, DocValuesTest, SolrCmdDistributorTest, ZkCLITest, TestSystemIdResolver, BigEndianAscendingWordDeserializerTest, TestBinaryResponseWriter, SolrIndexSplitterTest, TestNamedUpdateProcessors, DirectSolrSpellCheckerTest, SimplePostToolTest, TestNoOpRegenerator, SuggestComponentContextFilterQueryTest, TestSolrDeletionPolicy2, TestStressVersions, DistributedIntervalFacetingTest, UniqFieldsUpdateProcessorFactoryTest, TestConfigSetsAPIZkFailure, TestRandomMergePolicy, MigrateRouteKeyTest, ConcurrentDeleteAndCreateCollectionTest, TestUpdate, AssignTest, TestDFRSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestSQLHandler, ConvertedLegacyTest, JavabinLoaderTest, TestUseDocValuesAsStored2, TestExactSharedStatsCache, TestSchemaVersionResource, TestHdfsUpdateLog, OpenExchangeRatesOrgProviderTest, ZkStateWriterTest, UpdateRequestProcessorFactoryTest, TestRawResponseWriter, TestSortingResponseWriter, TestPKIAuthenticationPlugin, DistributedDebugComponentTest, RemoteQueryErrorTest, IndexBasedSpellCheckerTest, CachingDirectoryFactoryTest, TestInitQParser, CollectionStateFormat2Test, TermsComponentTest, TestSchemaNameResource, TestQueryUtils, DistributedFacetPivotWhiteBoxTest, BasicFunctionalityTest, ExplicitHLLTest, TestSolrCoreProperties, TestMacros, HdfsCollectionsAPIDistributedZkTest, TestRebalanceLeaders, TestScoreJoinQPNoScore, SchemaVersionSpecificBehaviorTest, SpellingQueryConverterTest, WordBreakSolrSpellCheckerTest, LeaderFailoverAfterPartitionTest, TestDynamicFieldResource, TestRequestStatusCollectionAPI, HighlighterTest, TriLevelCompositeIdRoutingTest, SoftAutoCommitTest, TestCopyFieldCollectionResource, DistributedSuggestComponentTest, TestNRTOpen, TestCollectionAPI, RequestLoggingTest, TestHighFrequencyDictionaryFactory, CSVRequestHandlerTest, SolrInfoMBeanTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestJettySolrRunner, ShardRoutingCustomTest, TestSolrQueryParserResource, TestCollapseQParserPlugin, CdcrRequestHandlerTest, NotRequiredUniqueKeyTest, TestBM25SimilarityFactory, TestDownShardTolerantSearch, TermVectorComponentTest, TestPartialUpdateDeduplication, DistributedTermsComponentTest, TestJsonFacets, DocumentBuilderTest, CustomCollectionTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed [295/573 (1!)] on J1 in 601.45s, 1 test, 1 error <<< FAILURES!

[...truncated 895 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:791: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:728: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:991: There were test failures: 573 suites (8 ignored), 2282 tests, 1 error, 33 ignored (18 assumptions) [seed: 8AF97C780B82D319]

Total time: 215 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 894 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/894/

2 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=14233, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=14233, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:53816: collection already exists: awholynewstresscollection_collection4_0
	at __randomizedtesting.SeedInfo.seed([97DE1102D84D2B94]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1100)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:871)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:807)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1574)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:881)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.HttpPartitionTest

Error Message:
ObjectTracker found 3 object(s) that were not released!!! [RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 3 object(s) that were not released!!! [RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper]
	at __randomizedtesting.SeedInfo.seed([97DE1102D84D2B94]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:229)
	at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10748 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/init-core-data-001
   [junit4]   2> 1546950 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[97DE1102D84D2B94]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1546950 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[97DE1102D84D2B94]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1546954 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1546954 INFO  (Thread-7664) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1546954 INFO  (Thread-7664) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1547054 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.ZkTestServer start zk server on port:50964
   [junit4]   2> 1547054 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1547055 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1547058 INFO  (zkCallback-1721-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@40355090 name:ZooKeeperConnection Watcher:127.0.0.1:50964 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1547058 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1547059 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1547059 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1547062 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1547072 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1547078 INFO  (zkCallback-1722-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1db6f600 name:ZooKeeperConnection Watcher:127.0.0.1:50964/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1547078 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1547078 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1547078 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1547081 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1547082 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1547084 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1547087 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1547087 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1547090 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1547090 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1547093 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1547094 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1547096 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1547096 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1547097 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1547098 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1547099 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1547099 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1547102 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1547102 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1547105 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1547105 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1547107 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1547107 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1547109 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1547109 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1547111 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1547111 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1547492 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1
   [junit4]   2> 1547494 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 1547496 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@64f3f7af{/,null,AVAILABLE}
   [junit4]   2> 1547497 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.e.j.s.ServerConnector Started ServerConnector@25cb4cc0{HTTP/1.1,[http/1.1]}{127.0.0.1:59511}
   [junit4]   2> 1547497 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.e.j.s.Server Started @1550643ms
   [junit4]   2> 1547497 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/tempDir-001/control/data, hostContext=/, hostPort=59511, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores}
   [junit4]   2> 1547497 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1547497 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001'
   [junit4]   2> 1547497 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1547497 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1547498 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1547500 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1547508 INFO  (zkCallback-1723-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1dc0aa33 name:ZooKeeperConnection Watcher:127.0.0.1:50964/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1547508 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1547509 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1547520 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1547520 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/solr.xml
   [junit4]   2> 1547525 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores
   [junit4]   2> 1547526 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.CoreContainer New CoreContainer 1733974655
   [junit4]   2> 1547526 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001]
   [junit4]   2> 1547526 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/lib
   [junit4]   2> 1547526 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1547527 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1547528 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1547528 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1547528 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50964/solr
   [junit4]   2> 1547528 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1547528 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1547537 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1547548 INFO  (zkCallback-1726-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7aa79a0d name:ZooKeeperConnection Watcher:127.0.0.1:50964 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1547548 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1547549 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1547552 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1547553 INFO  (zkCallback-1727-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@31d19590 name:ZooKeeperConnection Watcher:127.0.0.1:50964/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1547554 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1547555 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1547557 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1547559 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1547561 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1547563 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1547564 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1547566 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1547567 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1547568 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1547569 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59511_
   [junit4]   2> 1547570 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:59511_
   [junit4]   2> 1547571 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1547572 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1547573 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1547574 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95107023152742404-127.0.0.1:59511_-n_0000000000
   [junit4]   2> 1547574 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59511_
   [junit4]   2> 1547574 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1547576 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.Overseer Overseer (id=95107023152742404-127.0.0.1:59511_-n_0000000000) starting
   [junit4]   2> 1547579 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1547590 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1547590 INFO  (OverseerCollectionConfigSetProcessor-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1547590 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1547591 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1547602 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1547602 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1547603 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores
   [junit4]   2> 1547604 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1547604 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1
   [junit4]   2> 1547604 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1547605 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1547605 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:59511_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1547605 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1547606 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1547607 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:59511",
   [junit4]   2>   "node_name":"127.0.0.1:59511_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1547607 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1547608 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:59511",
   [junit4]   2>   "node_name":"127.0.0.1:59511_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1547608 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1547609 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1547610 INFO  (zkCallback-1727-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1548608 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1548608 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1548613 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1548613 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1548615 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1548615 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1'
   [junit4]   2> 1548615 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1548615 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1548617 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1548623 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1548628 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1548655 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1548662 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1548783 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1548793 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1548795 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1548810 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1548812 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1548814 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1548815 WARN  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1548815 WARN  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1548815 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1548816 WARN  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1548816 WARN  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1548819 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1548819 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1548821 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1548822 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1/data/]
   [junit4]   2> 1548822 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@497dfe92
   [junit4]   2> 1548822 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1/data
   [junit4]   2> 1548822 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1548822 WARN  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1548823 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1/data/index
   [junit4]   2> 1548823 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@25e4cf5b
   [junit4]   2> 1548823 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@791a6fdd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2c81f324),segFN=segments_1,generation=1}
   [junit4]   2> 1548824 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1548827 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1548827 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1548827 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1548827 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1548827 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1548828 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1548828 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1/data/
   [junit4]   2> 1548828 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1548828 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1548828 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1548828 WARN  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1548831 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548832 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548833 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548834 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1548837 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1548837 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1548838 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1548842 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1548842 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1548844 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1548845 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1548845 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6bb35391
   [junit4]   2> 1548846 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@791a6fdd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2c81f324),segFN=segments_1,generation=1}
   [junit4]   2> 1548846 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1548846 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@6be4007d[collection1] main
   [junit4]   2> 1548846 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1548847 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1548847 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1548847 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1548847 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1548847 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1548848 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1548848 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1548848 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1548848 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1548848 INFO  (searcherExecutor-3702-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6be4007d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1548849 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1548849 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1548849 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1548849 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1521712372428832768
   [junit4]   2> 1548851 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 1521712372428832768
   [junit4]   2> 1548851 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1548852 INFO  (coreLoadExecutor-3701-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1548852 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:59511 collection:control_collection shard:shard1
   [junit4]   2> 1548853 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1548856 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95107023152742404-core_node1-n_0000000000
   [junit4]   2> 1548856 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 1548858 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1548859 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1548859 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1548859 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:59511/collection1/
   [junit4]   2> 1548859 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1548859 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:59511/collection1/ has no replicas
   [junit4]   2> 1548863 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1548867 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node
   [junit4]   2> 1548869 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:59511",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1548870 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:59511/collection1/ shard1
   [junit4]   2> 1548971 INFO  (zkCallback-1727-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1549021 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:59511/collection1/ and leader is http://127.0.0.1:59511/collection1/
   [junit4]   2> 1549021 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59511
   [junit4]   2> 1549021 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1549021 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1549021 INFO  (coreZkRegister-3695-thread-1-processing-n:127.0.0.1:59511_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59511_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1549023 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:59511",
   [junit4]   2>   "node_name":"127.0.0.1:59511_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 1549023 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:59511",
   [junit4]   2>   "node_name":"127.0.0.1:59511_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1549107 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1549108 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1549109 INFO  (zkCallback-1729-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@39a676ba name:ZooKeeperConnection Watcher:127.0.0.1:50964/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1549109 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1549109 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1549109 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1549112 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1549112 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1549113 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1549113 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1549114 INFO  (zkCallback-1730-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ffa810c name:ZooKeeperConnection Watcher:127.0.0.1:50964/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1549115 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1549115 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1549121 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 2
   [junit4]   2> 1549121 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 1549121 INFO  (zkCallback-1729-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1549125 INFO  (zkCallback-1727-thread-1-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1549229 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 1549513 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/cores/collection1
   [junit4]   2> 1549514 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001
   [junit4]   2> 1549516 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 1549538 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2bc7b2f7{/,null,AVAILABLE}
   [junit4]   2> 1549538 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.e.j.s.ServerConnector Started ServerConnector@38f9c918{HTTP/1.1,[http/1.1]}{127.0.0.1:49679}
   [junit4]   2> 1549538 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.e.j.s.Server Started @1552685ms
   [junit4]   2> 1549538 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=49679, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/cores}
   [junit4]   2> 1549539 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 1549539 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001'
   [junit4]   2> 1549539 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1549539 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1549539 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1549540 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1549546 INFO  (zkCallback-1731-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4ca102b9 name:ZooKeeperConnection Watcher:127.0.0.1:50964/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1549546 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1549546 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1549561 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1549561 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/solr.xml
   [junit4]   2> 1549566 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/cores
   [junit4]   2> 1549566 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.CoreContainer New CoreContainer 2105025222
   [junit4]   2> 1549566 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001]
   [junit4]   2> 1549567 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/lib
   [junit4]   2> 1549567 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1549568 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1549569 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1549569 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1549569 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:50964/solr
   [junit4]   2> 1549569 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1549569 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1549574 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1549574 INFO  (zkCallback-1734-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1aaa700a name:ZooKeeperConnection Watcher:127.0.0.1:50964 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1549574 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1549575 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1549579 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1549580 INFO  (zkCallback-1735-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@596bcb0a name:ZooKeeperConnection Watcher:127.0.0.1:50964/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1549580 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1549584 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1550588 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49679_
   [junit4]   2> 1550589 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:49679_
   [junit4]   2> 1550590 INFO  (zkCallback-1729-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1550590 INFO  (zkCallback-1727-thread-2-processing-n:127.0.0.1:59511_) [n:127.0.0.1:59511_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1550590 INFO  (zkCallback-1735-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1550592 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1550592 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95107023152742409-127.0.0.1:49679_-n_0000000001
   [junit4]   2> 1550593 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95107023152742404-127.0.0.1:59511_-n_0000000000 to know if I could be the leader
   [junit4]   2> 1550603 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1550603 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1550604 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/cores
   [junit4]   2> 1550605 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1550605 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/cores/collection1
   [junit4]   2> 1550605 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1550607 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1550607 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[97DE1102D84D2B94]) [n:127.0.0.1:49679_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1550608 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1550608 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1550609 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.ZkController Registering watch for collection collection1
   [junit4]   2> 1550609 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [collection1]
   [junit4]   2> 1550609 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49679",
   [junit4]   2>   "node_name":"127.0.0.1:49679_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 1550609 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Add data for [collection1] ver [0]
   [junit4]   2> 1550610 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1550610 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49679",
   [junit4]   2>   "node_name":"127.0.0.1:49679_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1550610 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.o.ReplicaMutator Collection already exists with numShards=2
   [junit4]   2> 1550610 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1550712 INFO  (OverseerStateUpdate-95107023152742404-127.0.0.1:59511_-n_0000000000) [n:127.0.0.1:59511_    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 1550713 INFO  (zkCallback-1735-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1550713 INFO  (zkCallback-1735-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_    ] o.a.s.c.c.ZkStateReader Updating data for [collection1] from [0] to [1]
   [junit4]   2> 1551610 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1551610 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.ZkController Check for collection zkNode:collection1
   [junit4]   2> 1551610 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1551610 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/collection1]
   [junit4]   2> 1551611 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/collection1] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1551611 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/cores/collection1'
   [junit4]   2> 1551611 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1551611 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1551613 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1551617 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1551622 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1551635 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1551641 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1551768 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1551775 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1551776 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1551784 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1551787 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1551789 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1551789 WARN  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1551789 WARN  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1551790 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1551790 WARN  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Unknown key IMPORTANT NOTE
   [junit4]   2> 1551791 WARN  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Expected key, got STRING
   [junit4]   2> 1551792 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 1551792 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1551792 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1551792 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 1551792 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@497dfe92
   [junit4]   2> 1551793 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.cloud.CollectionsAPIDistributedZkTest_97DE1102D84D2B94-001/shard-1-001/cores/collection1/data
   [junit4]   2> 1551793 INFO  (coreLoadExecutor-3712-thread-1-processing-n:127.0.0.1:49679_) [n:127.0.0.1:49679_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/

[...truncated too long message...]

orUtil.java:232)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:352)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:160)
   [junit4]   2> 	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:412)
   [junit4]   2> 	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:225)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:352)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:160)
   [junit4]   2> 	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:412)
   [junit4]   2> 	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:225)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {range_facet_l_dv=BlockTreeOrds(blocksize=128), _version_=PostingsFormat(name=Memory doPackFST= false), multiDefault=PostingsFormat(name=SimpleText), a_t=PostingsFormat(name=Memory doPackFST= false), intDefault=PostingsFormat(name=Memory doPackFST= false), id=BlockTreeOrds(blocksize=128), range_facet_i_dv=PostingsFormat(name=Memory doPackFST= false), text=PostingsFormat(name=Direct), range_facet_l=PostingsFormat(name=Memory doPackFST= false), timestamp=PostingsFormat(name=Memory doPackFST= false)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene54), range_facet_i_dv=DocValuesFormat(name=Lucene54), timestamp=DocValuesFormat(name=Lucene54)}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=is_IS, timezone=PRT
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=4,threads=1,free=29813936,total=526909440
   [junit4]   2> NOTE: All tests run in this JVM: [HdfsChaosMonkeyNothingIsSafeTest, OverseerCollectionConfigSetProcessorTest, OverseerTest, TestRawTransformer, TestConfigSetsAPI, BadComponentTest, CdcrReplicationDistributedZkTest, HighlighterMaxOffsetTest, SSLMigrationTest, TestRandomCollapseQParserPlugin, LukeRequestHandlerTest, SmileWriterTest, RegexBoostProcessorTest, TestRecovery, TestShortCircuitedRequests, HardAutoCommitTest, TestCollationField, TestElisionMultitermQuery, TestIndexingPerformance, DirectSolrConnectionTest, TestSearchPerf, TestStressReorder, TestObjectReleaseTracker, BlockJoinFacetRandomTest, DistributedFacetPivotSmallAdvancedTest, TestInitQParser, TestSolrJ, ReplicationFactorTest, SolrTestCaseJ4Test, TestScoreJoinQPNoScore, HdfsNNFailoverTest, TestSchemaResource, SliceStateTest, BasicDistributedZk2Test, DateMathParserTest, OverseerTaskQueueTest, ScriptEngineTest, StatsComponentTest, StatelessScriptUpdateProcessorFactoryTest, TestChildDocTransformer, AddBlockUpdateTest, DeleteInactiveReplicaTest, TestManagedSchemaFieldTypeResource, TestBlobHandler, TestSuggestSpellingConverter, HighlighterConfigTest, DataDrivenBlockJoinTest, TestStressRecovery, VMParamsZkACLAndCredentialsProvidersTest, TestHighFrequencyDictionaryFactory, TestUtils, CdcrVersionReplicationTest, TestPKIAuthenticationPlugin, TestRecoveryHdfs, TestUniqueKeyFieldResource, TestCollapseQParserPlugin, CircularListTest, TestJsonFacets, TestFieldCollectionResource, TestPerFieldSimilarity, XmlUpdateRequestHandlerTest, TestSolrQueryParser, TestCryptoKeys, TestPivotHelperCode, TestReload, TestHdfsUpdateLog, TestTolerantSearch, TestConfigOverlay, DistributedQueryComponentCustomSortTest, TestSimpleQParserPlugin, TestFieldTypeResource, TestWriterPerf, CoreAdminHandlerTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestDefaultSearchFieldResource, SampleTest, SecurityConfHandlerTest, UnloadDistributedZkTest, BadCopyFieldTest, TestNamedUpdateProcessors, TestClusterStateMutator, TestJmxIntegration, TestExactSharedStatsCache, TestFastWriter, SolrCoreCheckLockOnStartupTest, ActionThrottleTest, AsyncCallRequestStatusResponseTest, CursorMarkTest, PluginInfoTest, NoCacheHeaderTest, TestCSVResponseWriter, BasicZkTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, ZkCLITest, ShardRoutingCustomTest, TestDistributedSearch, TestGroupingSearch, TestFiltering, SuggesterFSTTest, SuggesterTSTTest, SpatialFilterTest, TestCSVLoader, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, DirectUpdateHandlerOptimizeTest, TestRemoteStreaming, TestSolrDeletionPolicy1, StandardRequestHandlerTest, PrimitiveFieldTypeTest, RequestHandlersTest, PathHierarchyTokenizerFactoryTest, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, FastVectorHighlighterTest, LoggingHandlerTest, QueryParsingTest, JsonLoaderTest, SearchHandlerTest, UpdateParamsTest, TestSolrIndexConfig, SolrIndexConfigTest, OutputWriterTest, TestPostingsSolrHighlighter, TestCharFilters, TestCodecSupport, TestXIncludeConfig, TestDFRSimilarityFactory, TestJmxMonitoredMap, OpenExchangeRatesOrgProviderTest, PreAnalyzedFieldTest, PrimUtilsTest, ClusterStateTest, DistributedIntervalFacetingTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, CollectionStateFormat2Test, CollectionsAPIAsyncDistributedZkTest, ConfigSetsAPITest, CustomCollectionTest, HttpPartitionTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HttpPartitionTest -Dtests.seed=97DE1102D84D2B94 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=is_IS -Dtests.timezone=PRT -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J2 | HttpPartitionTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 3 object(s) that were not released!!! [RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([97DE1102D84D2B94]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:229)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [437/571 (2!)] on J2 in 175.03s, 1 test, 1 failure <<< FAILURES!

[...truncated 453 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:791: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:728: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:516: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1436: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:991: There were test failures: 571 suites (8 ignored), 2276 tests, 1 suite-level error, 1 error, 33 ignored (18 assumptions) [seed: 97DE1102D84D2B94]

Total time: 208 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any