You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/03 02:59:19 UTC
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1546
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1546/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC
All tests passed
Build Log:
[...truncated 11192 lines...]
[junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20140503_001903_822.syserr
[junit4] >>> JVM J0: stderr (verbatim) ----
[junit4] java(215,0x146169000) malloc: *** error for object 0x146157b00: pointer being freed was not allocated
[junit4] *** set a breakpoint in malloc_error_break to debug
[junit4] <<< JVM J0: EOF ----
[...truncated 1 lines...]
[junit4] ERROR: JVM J0 ended with an exception, command line: /Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/bin/java -XX:-UseCompressedOops -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/heapdumps -Dtests.prefix=tests -Dtests.seed=1E5E2F4D9463327A -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=5.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts.gracious=false -Dtests.multiplier=1 -DtempDir=. -Djava.io.tmpdir=. -Djunit4.tempDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp -Dclover.db.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/clover/db -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Djava.security.policy=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/tools/junit4/tests.policy -Dlucene.version=5.0-SNAPSHOT -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Dtests.disableHdfs=true -classpath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/classes/test:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-test-framework/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/test-framework/lib/junit4-ant-2.1.3.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/test-framework/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/codecs/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-solrj/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/analysis/common/lucene-analyzers-common-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/codecs/lucene-codecs-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/highlighter/lucene-highlighter-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/memory/lucene-memory-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/misc/lucene-misc-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/spatial/lucene-spatial-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/expressions/lucene-expressions-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/suggest/lucene-suggest-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/grouping/lucene-grouping-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/queries/lucene-queries-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/queryparser/lucene-queryparser-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/join/lucene-join-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/antlr-runtime-3.5.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/asm-4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/asm-commons-4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-cli-1.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-codec-1.9.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-configuration-1.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-fileupload-1.2.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-lang-2.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/concurrentlinkedhashmap-lru-1.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/dom4j-1.6.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/guava-14.0.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-annotations-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-auth-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-common-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-hdfs-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hppc-0.5.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/joda-time-2.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/log4j-1.2.16.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/org.restlet-2.1.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/org.restlet.ext.servlet-2.1.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/protobuf-java-2.5.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/slf4j-log4j12-1.7.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/spatial4j-0.4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/commons-io-2.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/httpclient-4.3.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/httpcore-4.3.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/httpmime-4.3.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/noggit-0.5.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/slf4j-api-1.7.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/wstx-asl-3.2.7.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/zookeeper-3.4.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-continuation-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-deploy-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-http-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-io-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-jmx-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-security-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-server-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-servlet-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-util-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-webapp-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-xml-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/servlet-api-3.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/core/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/test-framework/lib/junit-4.10.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/test-framework/lib/randomizedtesting-runner-2.1.3.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/cglib-nodep-2.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/commons-collections-3.2.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/easymock-3.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/hadoop-common-2.2.0-tests.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/hadoop-hdfs-2.2.0-tests.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/javax.servlet-api-3.0.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/jcl-over-slf4j-1.7.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/jersey-core-1.8.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/jetty-6.1.26.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/jetty-util-6.1.26.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/objenesis-1.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/analysis/icu/lucene-analyzers-icu-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-analysis-extras/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/contrib/analysis-extras/lib/icu4j-52.1.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-launcher.jar:/Users/jenkins/.ant/lib/ivy-2.3.0.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-antlr.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bcel.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bsf.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-log4j.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-oro.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-regexp.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-resolver.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-xalan2.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-logging.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-net.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jai.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-javamail.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jdepend.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jmf.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jsch.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit4.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-netrexx.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-swing.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-testutil.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/lib/tools.jar:/Users/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.1.3.jar -ea:org.apache.lucene... -ea:org.apache.solr... com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -flush -eventsfile /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20140503_001903_822.events @/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20140503_001903_822.suites
[junit4] ERROR: JVM J0 ended with an exception: Forked process returned with error code: 134. Very likely a JVM crash. Process output piped in logs above.
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1461)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:133)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:948)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:945)
[junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4] at java.lang.Thread.run(Thread.java:745)
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1296: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:920: At least one slave process threw an exception, first: Forked process returned with error code: 134. Very likely a JVM crash. Process output piped in logs above.
Total time: 77 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1548
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1548/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
expected:<0> but was:<1>
Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
at __randomizedtesting.SeedInfo.seed([75930B0030B69967:F475851847E9F95B]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:141)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11077 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/init-core-data-001
[junit4] 2> 1846890 T5908 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 1846891 T5908 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_v/n
[junit4] 2> 1846897 T5908 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 1846899 T5908 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1846900 T5909 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1847002 T5908 oasc.ZkTestServer.run start zk server on port:51071
[junit4] 2> 1847004 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1847011 T5915 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36b056ec name:ZooKeeperConnection Watcher:127.0.0.1:51071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1847012 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1847012 T5908 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1847021 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1847026 T5917 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11cfc762 name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1847027 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1847027 T5908 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1847034 T5908 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1847039 T5908 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1847043 T5908 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1847048 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1847049 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1847057 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 1847058 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1847065 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1847065 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1847070 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1847071 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1847077 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1847079 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1847084 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1847084 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1847091 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1847092 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 1847103 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1847103 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1847108 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1847109 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1847116 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1847116 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1847121 T5908 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1847122 T5908 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1847132 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1847137 T5919 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7433772c name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1847137 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1848837 T5908 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1848844 T5908 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1848846 T5908 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51075
[junit4] 2> 1848850 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1848851 T5908 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1848851 T5908 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002
[junit4] 2> 1848852 T5908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/'
[junit4] 2> 1848899 T5908 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/solr.xml
[junit4] 2> 1848928 T5908 oasc.CoreContainer.<init> New CoreContainer 2033464285
[junit4] 2> 1848928 T5908 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/]
[junit4] 2> 1848929 T5908 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1848930 T5908 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1848930 T5908 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1848931 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1848931 T5908 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1848931 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1848931 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1848932 T5908 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1848933 T5908 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1848937 T5908 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1848937 T5908 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1848938 T5908 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1848938 T5908 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51071/solr
[junit4] 2> 1848938 T5908 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1848941 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1848944 T5930 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61d1b192 name:ZooKeeperConnection Watcher:127.0.0.1:51071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1848945 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1848953 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1848955 T5932 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d184f21 name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1848956 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1848959 T5908 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1848967 T5908 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1848979 T5908 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 1848986 T5908 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 1848997 T5908 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 1849008 T5908 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1849011 T5908 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51075__v%2Fn
[junit4] 2> 1849013 T5908 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51075__v%2Fn
[junit4] 2> 1849019 T5908 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1849024 T5908 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1849034 T5908 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51075__v%2Fn
[junit4] 2> 1849034 T5908 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1849037 T5908 oasc.Overseer.start Overseer (id=91693664051200004-127.0.0.1:51075__v%2Fn-n_0000000000) starting
[junit4] 2> 1849044 T5908 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1849068 T5934 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1849072 T5934 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:51075__v%2Fn
[junit4] 2> 1849072 T5908 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1849079 T5908 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1849083 T5908 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1849094 T5933 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1849097 T5935 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 1849097 T5935 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1849099 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1849100 T5935 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1849102 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1849104 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:51075/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51075__v%2Fn",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1849104 T5933 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 1849104 T5933 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1849110 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1849112 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1850105 T5935 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1850106 T5935 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1850111 T5935 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1850112 T5935 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1850116 T5935 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1850116 T5935 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/collection1/'
[junit4] 2> 1850120 T5935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 1850121 T5935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 1850121 T5935 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 1850180 T5935 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 1850201 T5935 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1850205 T5935 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1850209 T5935 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1850376 T5935 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1850377 T5935 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1850378 T5935 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1850383 T5935 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1850435 T5935 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 1850435 T5935 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1850435 T5935 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/
[junit4] 2> 1850436 T5935 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fcc0205
[junit4] 2> 1850438 T5935 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data
[junit4] 2> 1850439 T5935 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/index/
[junit4] 2> 1850440 T5935 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1850441 T5935 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/index
[junit4] 2> 1850443 T5935 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=67.3037109375, floorSegmentMB=1.296875, forceMergeDeletesPctAllowed=18.754281475137972, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1850448 T5935 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1850448 T5935 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1850452 T5935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1850454 T5935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1850455 T5935 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1850455 T5935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1850456 T5935 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1850456 T5935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1850457 T5935 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1850457 T5935 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1850458 T5935 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1850459 T5935 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1850459 T5935 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1850459 T5935 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1850459 T5935 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1850460 T5935 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1850460 T5935 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1850461 T5935 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1850461 T5935 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1850470 T5935 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1850473 T5935 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1850473 T5935 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1850475 T5935 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=17, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7277011005546191]
[junit4] 2> 1850488 T5935 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1850488 T5935 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1850489 T5935 oass.SolrIndexSearcher.<init> Opening Searcher@235db3b4[collection1] main
[junit4] 2> 1850489 T5935 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1850491 T5935 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1850492 T5935 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1850494 T5935 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1850495 T5935 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1850495 T5935 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1850497 T5935 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1850497 T5935 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1850497 T5935 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 1850498 T5935 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1850500 T5936 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@235db3b4[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1850501 T5935 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1850503 T5939 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:51075/_v/n collection:control_collection shard:shard1
[junit4] 2> 1850506 T5908 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1850506 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1850512 T5939 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1850514 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1850520 T5941 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@363a5978 name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1850520 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1850525 T5908 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1850532 T5908 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1850558 T5939 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1850568 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850569 T5939 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1850569 T5939 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C3551 name=collection1 org.apache.solr.core.SolrCore@31a62f76 url=https://127.0.0.1:51075/_v/n/collection1 node=127.0.0.1:51075__v%2Fn C3551_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:51075/_v/n, core=collection1, node_name=127.0.0.1:51075__v%2Fn}
[junit4] 2> 1850570 T5939 C3551 P51075 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:51075/_v/n/collection1/
[junit4] 2> 1850570 T5939 C3551 P51075 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1850570 T5939 C3551 P51075 oasc.SyncStrategy.syncToMe https://127.0.0.1:51075/_v/n/collection1/ has no replicas
[junit4] 2> 1850571 T5939 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:51075/_v/n/collection1/ shard1
[junit4] 2> 1850571 T5939 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1850578 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1850594 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850605 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850623 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850628 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1850628 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1850709 T5939 oasc.ZkController.register We are https://127.0.0.1:51075/_v/n/collection1/ and leader is https://127.0.0.1:51075/_v/n/collection1/
[junit4] 2> 1850709 T5939 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:51075/_v/n
[junit4] 2> 1850710 T5939 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1850710 T5939 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 1850710 T5939 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1850716 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850720 T5939 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1850727 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1850736 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:51075/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51075__v%2Fn",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1850755 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1850869 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1850870 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1852320 T5908 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 1852321 T5908 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1852337 T5908 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1852340 T5908 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51079
[junit4] 2> 1852346 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1852346 T5908 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1852347 T5908 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003
[junit4] 2> 1852347 T5908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/'
[junit4] 2> 1852403 T5908 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/solr.xml
[junit4] 2> 1852428 T5908 oasc.CoreContainer.<init> New CoreContainer 1492392607
[junit4] 2> 1852429 T5908 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/]
[junit4] 2> 1852430 T5908 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1852430 T5908 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1852431 T5908 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1852432 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1852432 T5908 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1852432 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1852432 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1852433 T5908 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1852434 T5908 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1852439 T5908 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1852440 T5908 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1852441 T5908 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1852441 T5908 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51071/solr
[junit4] 2> 1852442 T5908 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1852444 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1852454 T5952 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3298ee98 name:ZooKeeperConnection Watcher:127.0.0.1:51071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1852454 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1852464 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1852468 T5954 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@188995ad name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1852469 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1852491 T5908 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1853500 T5908 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51079__v%2Fn
[junit4] 2> 1853502 T5908 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51079__v%2Fn
[junit4] 2> 1853535 T5955 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1853535 T5955 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1853540 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1853541 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1853542 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1853540 T5955 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1853551 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1853565 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:51079/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51079__v%2Fn",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1853565 T5933 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1853565 T5933 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1853573 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1853576 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1853577 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1853577 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1854546 T5955 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1854547 T5955 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1854548 T5955 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1854548 T5955 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1854550 T5955 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1854550 T5955 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/collection1/'
[junit4] 2> 1854551 T5955 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 1854552 T5955 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 1854552 T5955 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 1854670 T5955 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 1854695 T5955 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1854706 T5955 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1854720 T5955 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1854951 T5955 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1854952 T5955 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1854954 T5955 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1854959 T5955 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1855008 T5955 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1855008 T5955 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1855009 T5955 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/
[junit4] 2> 1855009 T5955 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fcc0205
[junit4] 2> 1855010 T5955 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1
[junit4] 2> 1855011 T5955 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/index/
[junit4] 2> 1855012 T5955 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1855014 T5955 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/index
[junit4] 2> 1855015 T5955 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=67.3037109375, floorSegmentMB=1.296875, forceMergeDeletesPctAllowed=18.754281475137972, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1855019 T5955 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1855019 T5955 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1855025 T5955 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1855026 T5955 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1855026 T5955 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1855027 T5955 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1855027 T5955 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1855027 T5955 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1855029 T5955 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1855029 T5955 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1855030 T5955 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1855030 T5955 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1855030 T5955 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1855031 T5955 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1855031 T5955 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1855032 T5955 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1855033 T5955 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1855033 T5955 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1855034 T5955 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1855042 T5955 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1855044 T5955 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1855044 T5955 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1855047 T5955 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=17, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7277011005546191]
[junit4] 2> 1855052 T5955 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1855053 T5955 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1855053 T5955 oass.SolrIndexSearcher.<init> Opening Searcher@4ad53d77[collection1] main
[junit4] 2> 1855054 T5955 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1855058 T5955 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1855058 T5955 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1855059 T5955 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1855059 T5955 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1855059 T5955 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1855061 T5955 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1855061 T5955 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1855061 T5955 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 1855062 T5955 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1855070 T5956 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ad53d77[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1855070 T5955 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1855072 T5959 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:51079/_v/n collection:collection1 shard:shard2
[junit4] 2> 1855074 T5908 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1855074 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1855082 T5959 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1855133 T5959 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1855154 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855159 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1855161 T5959 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1855162 T5959 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C3552 name=collection1 org.apache.solr.core.SolrCore@3f52c174 url=https://127.0.0.1:51079/_v/n/collection1 node=127.0.0.1:51079__v%2Fn C3552_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:51079/_v/n, core=collection1, node_name=127.0.0.1:51079__v%2Fn}
[junit4] 2> 1855162 T5959 C3552 P51079 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:51079/_v/n/collection1/
[junit4] 2> 1855162 T5959 C3552 P51079 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1855163 T5959 C3552 P51079 oasc.SyncStrategy.syncToMe https://127.0.0.1:51079/_v/n/collection1/ has no replicas
[junit4] 2> 1855163 T5959 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:51079/_v/n/collection1/ shard2
[junit4] 2> 1855163 T5959 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1855176 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855178 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855178 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855181 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855208 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855223 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1855234 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855342 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855342 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855343 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855375 T5959 oasc.ZkController.register We are https://127.0.0.1:51079/_v/n/collection1/ and leader is https://127.0.0.1:51079/_v/n/collection1/
[junit4] 2> 1855376 T5959 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:51079/_v/n
[junit4] 2> 1855376 T5959 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1855376 T5959 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1855376 T5959 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1855388 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855392 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855393 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855396 T5959 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1855405 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1855409 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:51079/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51079__v%2Fn",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1855428 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1855534 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855535 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1855535 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1856974 T5908 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 1856975 T5908 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1856982 T5908 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1856985 T5908 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51082
[junit4] 2> 1856989 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1856989 T5908 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1856990 T5908 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004
[junit4] 2> 1856990 T5908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/'
[junit4] 2> 1857059 T5908 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/solr.xml
[junit4] 2> 1857076 T5908 oasc.CoreContainer.<init> New CoreContainer 959363488
[junit4] 2> 1857076 T5908 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/]
[junit4] 2> 1857077 T5908 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1857078 T5908 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1857078 T5908 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1857078 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1857079 T5908 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1857079 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1857079 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1857080 T5908 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1857080 T5908 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1857083 T5908 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1857084 T5908 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1857084 T5908 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1857085 T5908 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51071/solr
[junit4] 2> 1857086 T5908 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1857099 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1857100 T5970 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c03ef40 name:ZooKeeperConnection Watcher:127.0.0.1:51071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1857101 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1857108 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1857111 T5972 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@728db7b5 name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1857112 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1857133 T5908 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1858145 T5908 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51082__v%2Fn
[junit4] 2> 1858155 T5908 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51082__v%2Fn
[junit4] 2> 1858177 T5973 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1858178 T5973 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1858179 T5973 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1858180 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1858180 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1858180 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1858184 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1858186 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:51082/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51082__v%2Fn",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1858187 T5933 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1858187 T5933 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1858191 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1858194 T5972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1858195 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1858197 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1858198 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859181 T5973 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1859182 T5973 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1859184 T5973 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1859184 T5973 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1859186 T5973 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1859187 T5973 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/collection1/'
[junit4] 2> 1859188 T5973 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 1859188 T5973 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 1859189 T5973 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 1859238 T5973 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 1859262 T5973 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1859266 T5973 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1859284 T5973 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1859456 T5973 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1859456 T5973 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1859458 T5973 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1859466 T5973 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1859532 T5973 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 1859532 T5973 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1859533 T5973 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/
[junit4] 2> 1859534 T5973 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@fcc0205
[junit4] 2> 1859535 T5973 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2
[junit4] 2> 1859536 T5973 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/index/
[junit4] 2> 1859537 T5973 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1859539 T5973 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/index
[junit4] 2> 1859539 T5973 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=27, maxMergedSegmentMB=67.3037109375, floorSegmentMB=1.296875, forceMergeDeletesPctAllowed=18.754281475137972, segmentsPerTier=35.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1859544 T5973 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1859545 T5973 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1859550 T5973 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1859550 T5973 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1859551 T5973 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1859551 T5973 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1859552 T5973 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1859553 T5973 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1859554 T5973 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1859555 T5973 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1859555 T5973 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1859556 T5973 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1859556 T5973 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1859556 T5973 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1859557 T5973 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1859557 T5973 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1859557 T5973 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1859558 T5973 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1859559 T5973 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 1859570 T5973 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1859573 T5973 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1859574 T5973 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1859575 T5973 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=17, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7277011005546191]
[junit4] 2> 1859576 T5973 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1859576 T5973 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1859577 T5973 oass.SolrIndexSearcher.<init> Opening Searcher@725d9238[collection1] main
[junit4] 2> 1859577 T5973 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1859580 T5973 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1859580 T5973 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 1859582 T5973 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1859582 T5973 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 1859583 T5973 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1859586 T5973 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 1859586 T5973 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1859586 T5973 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 1859587 T5973 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 1859591 T5974 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@725d9238[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1859594 T5973 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1859597 T5977 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:51082/_v/n collection:collection1 shard:shard3
[junit4] 2> 1859598 T5908 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1859600 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1859602 T5977 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 1859617 T5977 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 1859635 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859641 T5977 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1859642 T5977 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C3553 name=collection1 org.apache.solr.core.SolrCore@c8ec20e url=https://127.0.0.1:51082/_v/n/collection1 node=127.0.0.1:51082__v%2Fn C3553_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:51082/_v/n, core=collection1, node_name=127.0.0.1:51082__v%2Fn}
[junit4] 2> 1859643 T5977 C3553 P51082 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:51082/_v/n/collection1/
[junit4] 2> 1859645 T5977 C3553 P51082 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1859645 T5977 C3553 P51082 oasc.SyncStrategy.syncToMe https://127.0.0.1:51082/_v/n/collection1/ has no replicas
[junit4] 2> 1859646 T5977 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:51082/_v/n/collection1/ shard3
[junit4] 2> 1859646 T5977 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 1859650 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1859664 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859675 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859685 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859791 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859792 T5972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859792 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859792 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859830 T5977 oasc.ZkController.register We are https://127.0.0.1:51082/_v/n/collection1/ and leader is https://127.0.0.1:51082/_v/n/collection1/
[junit4] 2> 1859830 T5977 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:51082/_v/n
[junit4] 2> 1859830 T5977 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1859831 T5977 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1859831 T5977 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1859833 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859833 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859834 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859834 T5977 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1859837 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1859839 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:51082/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51082__v%2Fn",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 1859845 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1859951 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859952 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859952 T5972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1859952 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1861255 T5908 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 1861256 T5908 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1861264 T5908 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 1861266 T5908 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51085
[junit4] 2> 1861270 T5908 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1861271 T5908 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1861271 T5908 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-005
[junit4] 2> 1861271 T5908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-005/'
[junit4] 2> 1861335 T5908 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-005/solr.xml
[junit4] 2> 1861364 T5908 oasc.CoreContainer.<init> New CoreContainer 2132999269
[junit4] 2> 1861365 T5908 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-005/]
[junit4] 2> 1861366 T5908 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1861367 T5908 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1861367 T5908 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1861367 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1861368 T5908 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1861369 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1861369 T5908 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1861369 T5908 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1861370 T5908 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1861376 T5908 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1861377 T5908 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1861377 T5908 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1861378 T5908 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51071/solr
[junit4] 2> 1861378 T5908 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1861381 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1861385 T5988 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54750aa7 name:ZooKeeperConnection Watcher:127.0.0.1:51071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1861385 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1861393 T5908 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1861398 T5990 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@525f9ace name:ZooKeeperConnection Watcher:127.0.0.1:51071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1861399 T5908 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1861447 T5908 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1862455 T5908 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51085__v%2Fn
[junit4] 2> 1862465 T5908 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51085__v%2Fn
[junit4] 2> 1862544 T5991 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1862544 T5991 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1862548 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1862548 T5991 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1862548 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1862549 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1862553 T5933 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1862555 T5933 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:51085/_v/n",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:51085__v%2Fn",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1862555 T5933 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1862555 T5933 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1862562 T5932 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1862564 T5972 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1862565 T5954 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1862565 T5941 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1862565 T5932 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1862565 T5990 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1863550 T5991 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1863551 T5991 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1863552 T5991 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1863553 T5991 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1863556 T5991 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1863556 T5991 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafe
[...truncated too long message...]
=segments_6,generation=6}
[junit4] 2> 2019685 T6794 C4312 P51122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 6
[junit4] 2> 2019699 T6794 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 2019700 T6794 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 2019701 T6794 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10;done=false>>]
[junit4] 2> 2019701 T6794 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10
[junit4] 2> 2019701 T6794 oasc.CachingDirectoryFactory.closeCacheValue looking to close /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10/index.20140504012554990 [CachedDir<<refCount=0;path=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10/index.20140504012554990;done=false>>]
[junit4] 2> 2019702 T6794 oasc.CachingDirectoryFactory.close Closing directory: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001/tempDir-001/jetty10/index.20140504012554990
[junit4] 2> 2019702 T6794 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 2019704 T6794 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-75930B0030B69967-001
[junit4] 2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=443), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=443)), sim=DefaultSimilarity, locale=nl_NL, timezone=ROK
[junit4] 2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=2,threads=1,free=65089976,total=317382656
[junit4] 2> NOTE: All tests run in this JVM: [TestRTGBase, SpellCheckCollatorTest, IndexSchemaTest, PathHierarchyTokenizerFactoryTest, AddSchemaFieldsUpdateProcessorFactoryTest, AtomicUpdatesTest, ShowFileRequestHandlerTest, TestReloadAndDeleteDocs, TestSweetSpotSimilarityFactory, SolrXmlInZkTest, TestOverriddenPrefixQueryForCustomFieldType, TestQuerySenderListener, TermVectorComponentDistributedTest, TestIndexSearcher, HdfsBasicDistributedZk2Test, AliasIntegrationTest, SuggestComponentTest, OpenCloseCoreStressTest, ShardSplitTest, AnalysisErrorHandlingTest, SimplePostToolTest, TestReload, CustomCollectionTest, BadIndexSchemaTest, TestUniqueKeyFieldResource, TestComplexPhraseQParserPlugin, PluginInfoTest, TestCloudInspectUtil, UnloadDistributedZkTest, TestAddFieldRealTimeGet, HdfsWriteToMultipleCollectionsTest, TestFoldingMultitermQuery, ReturnFieldsTest, SolrInfoMBeanTest, SliceStateUpdateTest, QueryParsingTest, BasicDistributedZkTest, TestCoreDiscovery, SearchHandlerTest, TestCustomSort, ClusterStateUpdateTest, TestWordDelimiterFilterFactory, SolrCoreCheckLockOnStartupTest, CachingDirectoryFactoryTest, TestRestManager, RequiredFieldsTest, DistribCursorPagingTest, SolrIndexSplitterTest, TestSolrQueryParserDefaultOperatorResource, ShardRoutingCustomTest, FieldMutatingUpdateProcessorTest, SOLR749Test, TestSerializedLuceneMatchVersion, TestDefaultSearchFieldResource, TestLMJelinekMercerSimilarityFactory, TestSolrDeletionPolicy2, DocumentAnalysisRequestHandlerTest, TestInitQParser, AbstractAnalyticsStatsTest, TestSchemaVersionResource, AddBlockUpdateTest, TestSurroundQueryParser, TestRangeQuery, TestManagedStopFilterFactory, CursorMarkTest, TestMiniSolrCloudCluster, TestJoin, DocValuesMultiTest, CursorPagingTest, TestRandomDVFaceting, SampleTest, RemoteQueryErrorTest, TestDFRSimilarityFactory, TestBM25SimilarityFactory, TestDistributedSearch, SchemaVersionSpecificBehaviorTest, ParsingFieldUpdateProcessorsTest, AnalysisAfterCoreReloadTest, TestSearcherReuse, BadComponentTest, TestFastOutputStream, TestSolrDeletionPolicy1, TestPartialUpdateDeduplication, TestHighlightDedupGrouping, TestLFUCache, ShardRoutingTest, TestGroupingSearch, NoFacetTest, TestTrie, TestCodecSupport, FullSolrCloudDistribCmdsTest, MinimalSchemaTest, HighlighterTest, TestShardHandlerFactory, BasicZkTest, TestPseudoReturnFields, HdfsBasicDistributedZkTest, TestBlendedInfixSuggestions, DistributedQueryElevationComponentTest, TestArbitraryIndexDir, TestSchemaSimilarityResource, InfoHandlerTest, TestFieldSortValues, FieldAnalysisRequestHandlerTest, BlockDirectoryTest, TestModifyConfFiles, DistribDocExpirationUpdateProcessorTest, IndexBasedSpellCheckerTest, HdfsChaosMonkeySafeLeaderTest, ResourceLoaderTest, XmlUpdateRequestHandlerTest, TestPerFieldSimilarity, ScriptEngineTest, ClusterStateTest, TestBadConfig, TestManagedSynonymFilterFactory, SimpleFacetsTest, SolrPluginUtilsTest, TestNoOpRegenerator, TimeZoneUtilsTest, TestXIncludeConfig, TestCloudManagedSchemaAddField, TestRequestStatusCollectionAPI, JsonLoaderTest, DistributedDebugComponentTest, TestDynamicFieldCollectionResource, BasicDistributedZk2Test, NoCacheHeaderTest, TestCollapseQParserPlugin, DistributedQueryComponentOptimizationTest, FastVectorHighlighterTest, RecoveryZkTest, TestSolrQueryParser, OpenExchangeRatesOrgProviderTest, SolrTestCaseJ4Test, NumericFieldsTest, TestManagedResourceStorage, SpellCheckComponentTest, ChangedSchemaMergeTest, CoreAdminHandlerTest, FunctionTest, TestRealTimeGet, SortByFunctionTest, TestSolrXMLSerializer, DistributedSuggestComponentTest, CSVRequestHandlerTest, DocValuesTest, TestRecoveryHdfs, TestSolrXmlPersistor, RegexBoostProcessorTest, TestFiltering, OutputWriterTest, TestPHPSerializedResponseWriter, SpellingQueryConverterTest, TriLevelCompositeIdRoutingTest, RequestHandlersTest, RangeFacetTest, RAMDirectoryFactoryTest, DirectUpdateHandlerOptimizeTest, TestSearchPerf, BinaryUpdateRequestHandlerTest, SpatialFilterTest, QueryEqualityTest, TestDistributedGrouping, EchoParamsTest, TestCursorMarkWithoutUniqueKey, TestQueryTypes, TestConfigSets, BasicFunctionalityTest, TestValueSourceCache, MoreLikeThisHandlerTest, CurrencyFieldOpenExchangeTest, TestLuceneMatchVersion, TestDocSet, CacheHeaderTest, DebugComponentTest, TestUtils, DeleteInactiveReplicaTest, HdfsLockFactoryTest, TestComponentsName, NotRequiredUniqueKeyTest, TestSolrXmlPersistence, FieldFacetTest, TestStressReorder, HdfsDirectoryTest, TestFieldCollectionResource, SuggesterTSTTest, AutoCommitTest, TestInfoStreamLogging, QueryResultKeyTest, TestIBSimilarityFactory, LoggingHandlerTest, ChaosMonkeySafeLeaderTest]
[junit4] Completed in 173.17s, 1 test, 1 failure <<< FAILURES!
[...truncated 618 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1296: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:920: There were test failures: 398 suites, 1670 tests, 1 failure, 42 ignored (15 assumptions)
Total time: 108 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1547
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1547/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC
2 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 176 from https://127.0.0.1:49482/collection1lastClient and got 175 from https://127.0.0.1:49491/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 176 from https://127.0.0.1:49482/collection1lastClient and got 175 from https://127.0.0.1:49491/collection1
at __randomizedtesting.SeedInfo.seed([FE214AABDDF27F5E:7FC7C4B3AAAD1F62]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1130)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1109)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
REGRESSION: org.apache.solr.cloud.OverseerStatusTest.testDistribSearch
Error Message:
reloadcollection the collection time out:180s
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: reloadcollection the collection time out:180s
at __randomizedtesting.SeedInfo.seed([FE214AABDDF27F5E:7FC7C4B3AAAD1F62]:0)
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
at org.apache.solr.cloud.OverseerStatusTest.invokeCollectionApi(OverseerStatusTest.java:142)
at org.apache.solr.cloud.OverseerStatusTest.doTest(OverseerStatusTest.java:103)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:793)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:453)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10534 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/init-core-data-001
[junit4] 2> 278560 T986 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 278561 T986 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 278570 T986 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 278574 T986 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 278575 T987 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 278675 T986 oasc.ZkTestServer.run start zk server on port:49474
[junit4] 2> 278676 T986 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 278684 T993 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c5f75d1 name:ZooKeeperConnection Watcher:127.0.0.1:49474 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 278684 T986 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 278685 T986 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 278696 T986 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 278700 T995 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@148dcff6 name:ZooKeeperConnection Watcher:127.0.0.1:49474/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 278700 T986 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 278700 T986 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 278706 T986 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 278711 T986 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 278717 T986 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 278723 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 278724 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 278734 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 278734 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 278739 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 278739 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 278744 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 278744 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 278749 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 278749 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 278754 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 278754 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 278760 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 278760 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 278765 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 278766 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 278771 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 278772 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 278777 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 278778 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 278783 T986 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 278784 T986 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 278791 T986 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 278809 T997 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32e40090 name:ZooKeeperConnection Watcher:127.0.0.1:49474/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 278810 T986 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 280514 T986 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 281149 T986 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 281170 T986 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49478
[junit4] 2> 281173 T986 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 281174 T986 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 281175 T986 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-002
[junit4] 2> 281175 T986 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-002/'
[junit4] 2> 281213 T986 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-002/solr.xml
[junit4] 2> 281257 T986 oasc.CoreContainer.<init> New CoreContainer 1568475567
[junit4] 2> 281257 T986 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-002/]
[junit4] 2> 281259 T986 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 281260 T986 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 281260 T986 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 281260 T986 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 281260 T986 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 281260 T986 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 281260 T986 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 281261 T986 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 281261 T986 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 281265 T986 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 281266 T986 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 281266 T986 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 281266 T986 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49474/solr
[junit4] 2> 281272 T986 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 281274 T986 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 281280 T1008 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54e155a9 name:ZooKeeperConnection Watcher:127.0.0.1:49474 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 281281 T986 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 281288 T986 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 281293 T1010 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4077ef61 name:ZooKeeperConnection Watcher:127.0.0.1:49474/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 281293 T986 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 281298 T986 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 281305 T986 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 281310 T986 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 281318 T986 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 281326 T986 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 281336 T986 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 281340 T986 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49478_
[junit4] 2> 281341 T986 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49478_
[junit4] 2> 281348 T986 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 281354 T986 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 281365 T986 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49478_
[junit4] 2> 281366 T986 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 281371 T986 oasc.Overseer.start Overseer (id=91691606601170948-127.0.0.1:49478_-n_0000000000) starting
[junit4] 2> 281378 T986 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 281406 T1012 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 281409 T1012 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:49478_
[junit4] 2> 281409 T986 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 281416 T986 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 281422 T986 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 281432 T1011 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 281435 T1013 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 281436 T1013 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 281438 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 281438 T1013 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 281442 T1011 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 281443 T1011 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:49478",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49478_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 281443 T1011 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 281444 T1011 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 281450 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 281452 T1010 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 282440 T1013 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 282441 T1013 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 282444 T1013 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 282444 T1013 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 282451 T1013 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 282451 T1013 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-002/collection1/'
[junit4] 2> 282454 T1013 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 282455 T1013 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 282456 T1013 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 282505 T1013 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 282537 T1013 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 282544 T1013 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 282557 T1013 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 282713 T1013 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 282715 T1013 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 282716 T1013 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 282721 T1013 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 282830 T1013 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 282830 T1013 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 282894 T1013 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/control/data/
[junit4] 2> 282894 T1013 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@18db59c8
[junit4] 2> 282896 T1013 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/control/data
[junit4] 2> 282897 T1013 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/control/data/index/
[junit4] 2> 282898 T1013 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 282899 T1013 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/control/data/index
[junit4] 2> 282899 T1013 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=37, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6029393882439132]
[junit4] 2> 282916 T1013 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 282916 T1013 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 282922 T1013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 282923 T1013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 282923 T1013 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 282923 T1013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 282923 T1013 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 282924 T1013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 282924 T1013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 282925 T1013 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 282925 T1013 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 282926 T1013 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 282926 T1013 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 282929 T1013 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 282930 T1013 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 282931 T1013 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 282932 T1013 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 282932 T1013 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 282932 T1013 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 282942 T1013 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 282946 T1013 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 282946 T1013 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 282947 T1013 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=44, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.38899710067071636]
[junit4] 2> 282948 T1013 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 282948 T1013 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 282949 T1013 oass.SolrIndexSearcher.<init> Opening Searcher@10c4fa11[collection1] main
[junit4] 2> 282949 T1013 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 282957 T1013 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 282958 T1013 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 282958 T1013 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 282959 T1013 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 282959 T1013 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 282960 T1013 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 282960 T1013 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 282960 T1013 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 282960 T1013 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 282966 T1014 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10c4fa11[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 282967 T1013 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 282969 T1017 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:49478 collection:control_collection shard:shard1
[junit4] 2> 282971 T986 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 282971 T986 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 282975 T1017 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 282983 T986 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 282999 T1019 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4567e604 name:ZooKeeperConnection Watcher:127.0.0.1:49474/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 282999 T986 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 283005 T986 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 283015 T986 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 283043 T1017 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 283052 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 283056 T1017 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 283056 T1017 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C280 name=collection1 org.apache.solr.core.SolrCore@3af58f60 url=https://127.0.0.1:49478/collection1 node=127.0.0.1:49478_ C280_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:49478, core=collection1, node_name=127.0.0.1:49478_}
[junit4] 2> 283056 T1017 C280 P49478 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:49478/collection1/
[junit4] 2> 283057 T1017 C280 P49478 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 283057 T1017 C280 P49478 oasc.SyncStrategy.syncToMe https://127.0.0.1:49478/collection1/ has no replicas
[junit4] 2> 283057 T1017 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:49478/collection1/ shard1
[junit4] 2> 283057 T1017 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 283061 T1011 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 283075 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 283079 T1019 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 283079 T1010 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 283104 T1011 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 283116 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 283230 T1019 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 283230 T1010 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 283251 T1017 oasc.ZkController.register We are https://127.0.0.1:49478/collection1/ and leader is https://127.0.0.1:49478/collection1/
[junit4] 2> 283251 T1017 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:49478
[junit4] 2> 283251 T1017 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 283251 T1017 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 283251 T1017 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 283258 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 283259 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 283259 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 283264 T1017 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 283268 T1011 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 283271 T1011 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:49478",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49478_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 283285 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 283402 T1019 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 283402 T1010 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 284791 T986 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 284793 T986 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 284814 T986 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 284817 T986 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49482
[junit4] 2> 284822 T986 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 284822 T986 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 284822 T986 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-003
[junit4] 2> 284823 T986 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-003/'
[junit4] 2> 284864 T986 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-003/solr.xml
[junit4] 2> 284884 T986 oasc.CoreContainer.<init> New CoreContainer 33545168
[junit4] 2> 284884 T986 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-003/]
[junit4] 2> 284885 T986 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 284885 T986 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 284885 T986 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 284885 T986 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 284886 T986 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 284886 T986 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 284886 T986 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 284887 T986 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 284887 T986 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 284892 T986 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 284893 T986 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 284893 T986 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 284894 T986 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49474/solr
[junit4] 2> 284894 T986 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 284896 T986 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 284908 T1030 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5829078c name:ZooKeeperConnection Watcher:127.0.0.1:49474 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 284910 T986 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 284919 T986 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 284925 T1032 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c1cebc5 name:ZooKeeperConnection Watcher:127.0.0.1:49474/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 284926 T986 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 284941 T986 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 285956 T986 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49482_
[junit4] 2> 285961 T986 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49482_
[junit4] 2> 286001 T1033 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 286001 T1033 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 286003 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 286003 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 286003 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 286004 T1033 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 286006 T1011 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 286008 T1011 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:49482",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49482_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 286008 T1011 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 286009 T1011 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 286015 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 286017 T1032 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 286017 T1019 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 286017 T1010 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 287006 T1033 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 287007 T1033 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 287012 T1033 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 287012 T1033 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 287020 T1033 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 287020 T1033 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-003/collection1/'
[junit4] 2> 287023 T1033 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 287025 T1033 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 287028 T1033 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 287087 T1033 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 287116 T1033 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 287119 T1033 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 287125 T1033 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 287273 T1033 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 287278 T1033 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 287279 T1033 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 287287 T1033 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 287345 T1033 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 287346 T1033 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 287346 T1033 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty1/
[junit4] 2> 287346 T1033 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@18db59c8
[junit4] 2> 287347 T1033 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty1
[junit4] 2> 287348 T1033 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty1/index/
[junit4] 2> 287348 T1033 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 287351 T1033 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty1/index
[junit4] 2> 287352 T1033 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=37, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6029393882439132]
[junit4] 2> 287359 T1033 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 287359 T1033 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 287372 T1033 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 287372 T1033 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 287372 T1033 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 287373 T1033 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 287373 T1033 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 287374 T1033 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 287375 T1033 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 287375 T1033 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 287375 T1033 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 287376 T1033 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 287376 T1033 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 287376 T1033 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 287377 T1033 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 287377 T1033 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 287378 T1033 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 287378 T1033 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 287378 T1033 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 287385 T1033 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 287388 T1033 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 287388 T1033 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 287390 T1033 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=44, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.38899710067071636]
[junit4] 2> 287391 T1033 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 287391 T1033 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 287391 T1033 oass.SolrIndexSearcher.<init> Opening Searcher@380bbdd6[collection1] main
[junit4] 2> 287397 T1033 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 287400 T1033 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 287400 T1033 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 287401 T1033 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 287402 T1033 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 287402 T1033 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 287403 T1033 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 287403 T1033 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 287403 T1033 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 287404 T1033 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 287407 T1034 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@380bbdd6[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 287410 T1033 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 287412 T1037 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:49482 collection:collection1 shard:shard2
[junit4] 2> 287414 T986 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 287414 T986 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 287415 T1037 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 287429 T1037 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 287434 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 287435 T1037 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 287435 T1037 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C281 name=collection1 org.apache.solr.core.SolrCore@6c650d96 url=https://127.0.0.1:49482/collection1 node=127.0.0.1:49482_ C281_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:49482, core=collection1, node_name=127.0.0.1:49482_}
[junit4] 2> 287435 T1037 C281 P49482 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:49482/collection1/
[junit4] 2> 287436 T1037 C281 P49482 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 287436 T1037 C281 P49482 oasc.SyncStrategy.syncToMe https://127.0.0.1:49482/collection1/ has no replicas
[junit4] 2> 287437 T1037 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:49482/collection1/ shard2
[junit4] 2> 287437 T1037 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 287447 T1011 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 287455 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 287460 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 287470 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 287584 T1010 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 287585 T1032 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 287585 T1019 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 287613 T1037 oasc.ZkController.register We are https://127.0.0.1:49482/collection1/ and leader is https://127.0.0.1:49482/collection1/
[junit4] 2> 287614 T1037 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:49482
[junit4] 2> 287615 T1037 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 287615 T1037 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 287615 T1037 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 287624 T1037 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 287624 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 287625 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 287625 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 287630 T1011 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 287632 T1011 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:49482",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49482_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 287639 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 287749 T1019 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 287749 T1032 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 287749 T1010 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 289075 T986 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 289077 T986 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 289091 T986 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 289093 T986 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49485
[junit4] 2> 289096 T986 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 289096 T986 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 289097 T986 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-004
[junit4] 2> 289097 T986 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-004/'
[junit4] 2> 289138 T986 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-004/solr.xml
[junit4] 2> 289162 T986 oasc.CoreContainer.<init> New CoreContainer 2100835746
[junit4] 2> 289163 T986 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-004/]
[junit4] 2> 289163 T986 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 289164 T986 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 289164 T986 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 289164 T986 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 289164 T986 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 289164 T986 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 289165 T986 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 289165 T986 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 289165 T986 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 289171 T986 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 289171 T986 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 289171 T986 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 289172 T986 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49474/solr
[junit4] 2> 289172 T986 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 289174 T986 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 289179 T1048 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e938da6 name:ZooKeeperConnection Watcher:127.0.0.1:49474 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 289180 T986 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 289187 T986 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 289191 T1050 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@607ae9e0 name:ZooKeeperConnection Watcher:127.0.0.1:49474/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 289192 T986 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 289213 T986 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 290226 T986 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49485_
[junit4] 2> 290232 T986 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49485_
[junit4] 2> 290269 T1051 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 290269 T1051 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 290271 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 290272 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 290272 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 290271 T1051 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 290276 T1011 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 290277 T1011 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:49485",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49485_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 290278 T1011 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 290278 T1011 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 290283 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 290286 T1032 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 290286 T1019 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 290287 T1010 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 290289 T1050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 291274 T1051 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 291275 T1051 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 291278 T1051 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 291278 T1051 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 291283 T1051 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 291284 T1051 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-004/collection1/'
[junit4] 2> 291287 T1051 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 291289 T1051 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 291290 T1051 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 291352 T1051 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 291366 T1051 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 291370 T1051 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 291375 T1051 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 291537 T1051 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 291539 T1051 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 291541 T1051 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 291545 T1051 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 291605 T1051 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 291608 T1051 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 291608 T1051 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty2/
[junit4] 2> 291608 T1051 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@18db59c8
[junit4] 2> 291609 T1051 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty2
[junit4] 2> 291610 T1051 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty2/index/
[junit4] 2> 291611 T1051 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 291612 T1051 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty2/index
[junit4] 2> 291612 T1051 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=37, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6029393882439132]
[junit4] 2> 291617 T1051 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 291617 T1051 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 291621 T1051 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 291622 T1051 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 291622 T1051 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 291622 T1051 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 291623 T1051 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 291623 T1051 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 291623 T1051 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 291624 T1051 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 291624 T1051 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 291625 T1051 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 291627 T1051 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 291627 T1051 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 291628 T1051 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 291628 T1051 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 291629 T1051 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 291630 T1051 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 291631 T1051 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 291638 T1051 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 291651 T1051 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 291651 T1051 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 291658 T1051 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=44, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.38899710067071636]
[junit4] 2> 291661 T1051 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 291661 T1051 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 291661 T1051 oass.SolrIndexSearcher.<init> Opening Searcher@4c9df948[collection1] main
[junit4] 2> 291662 T1051 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 291665 T1051 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 291665 T1051 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 291666 T1051 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 291666 T1051 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 291667 T1051 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 291668 T1051 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 291669 T1051 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 291669 T1051 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 291669 T1051 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 291672 T1052 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c9df948[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 291675 T1051 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 291676 T1055 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:49485 collection:collection1 shard:shard3
[junit4] 2> 291677 T986 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 291678 T986 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 291681 T1055 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 291713 T1055 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 291717 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291719 T1055 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 291719 T1055 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C282 name=collection1 org.apache.solr.core.SolrCore@5f1c4366 url=https://127.0.0.1:49485/collection1 node=127.0.0.1:49485_ C282_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:49485, core=collection1, node_name=127.0.0.1:49485_}
[junit4] 2> 291720 T1055 C282 P49485 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:49485/collection1/
[junit4] 2> 291720 T1055 C282 P49485 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 291720 T1055 C282 P49485 oasc.SyncStrategy.syncToMe https://127.0.0.1:49485/collection1/ has no replicas
[junit4] 2> 291721 T1055 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:49485/collection1/ shard3
[junit4] 2> 291721 T1055 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 291723 T1011 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 291732 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291737 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291747 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291853 T1032 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 291853 T1019 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 291853 T1010 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 291853 T1050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 291893 T1055 oasc.ZkController.register We are https://127.0.0.1:49485/collection1/ and leader is https://127.0.0.1:49485/collection1/
[junit4] 2> 291893 T1055 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:49485
[junit4] 2> 291893 T1055 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 291894 T1055 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 291894 T1055 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 291897 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291898 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291898 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 291899 T1055 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 291902 T1011 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 291903 T1011 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:49485",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:49485_",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 291909 T1010 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 292017 T1050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 292017 T1019 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 292017 T1032 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 292018 T1010 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 293341 T986 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 293342 T986 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 293363 T986 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 293370 T986 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:49488
[junit4] 2> 293382 T986 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 293383 T986 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 293383 T986 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-005
[junit4] 2> 293383 T986 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-005/'
[junit4] 2> 293428 T986 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-005/solr.xml
[junit4] 2> 293462 T986 oasc.CoreContainer.<init> New CoreContainer 212681335
[junit4] 2> 293463 T986 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-FE214AABDDF27F5E-001/tempDir-005/]
[junit4] 2> 293464 T986 oash
[...truncated too long message...]
2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:303)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.OverseerStatusTest-FE214AABDDF27F5E-001
[junit4] 2> NOTE: test params are: codec=Lucene45, sim=DefaultSimilarity, locale=es_PR, timezone=Atlantic/South_Georgia
[junit4] 2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=2,threads=1,free=107915304,total=466063360
[junit4] 2> NOTE: All tests run in this JVM: [TestFastOutputStream, HdfsBasicDistributedZk2Test, BinaryUpdateRequestHandlerTest, EchoParamsTest, TermVectorComponentDistributedTest, TestExpandComponent, SliceStateTest, TestManagedSynonymFilterFactory, SuggestComponentTest, TestElisionMultitermQuery, TestJmxIntegration, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, TestLFUCache, TestSolrXml, ClusterStateTest, AutoCommitTest, TestModifyConfFiles, CoreContainerCoreInitFailuresTest, TestSolrQueryParserDefaultOperatorResource, CollectionsAPIDistributedZkTest, TestSurroundQueryParser, TestCloudManagedSchema, TestLuceneMatchVersion, HighlighterTest, TestXIncludeConfig, FieldAnalysisRequestHandlerTest, DistributedDebugComponentTest, ChaosMonkeySafeLeaderTest, TestDocBasedVersionConstraints, TestAnalyzedSuggestions, MoreLikeThisHandlerTest, TestSuggestSpellingConverter, AlternateDirectoryTest, TestSolrIndexConfig, DirectUpdateHandlerOptimizeTest, EnumFieldTest, TestBlendedInfixSuggestions, DateFieldTest, SpellingQueryConverterTest, TestDefaultSimilarityFactory, TestRandomFaceting, AnalysisAfterCoreReloadTest, SpellPossibilityIteratorTest, TestPostingsSolrHighlighter, UpdateRequestProcessorFactoryTest, TestFuzzyAnalyzedSuggestions, CustomCollectionTest, DirectUpdateHandlerTest, TestShortCircuitedRequests, TestIBSimilarityFactory, RangeFacetTest, DocumentBuilderTest, DocExpirationUpdateProcessorFactoryTest, MigrateRouteKeyTest, ShardRoutingTest, TestSolr4Spatial, CircularListTest, FieldFacetTest, CoreAdminRequestStatusTest, RAMDirectoryFactoryTest, ShowFileRequestHandlerTest, BlockDirectoryTest, TestCopyFieldCollectionResource, SpellCheckCollatorTest, TestPerFieldSimilarity, TestSearchPerf, TestMultiCoreConfBootstrap, StressHdfsTest, CachingDirectoryFactoryTest, TestMiniSolrCloudCluster, TestRecovery, BasicDistributedZkTest, TestCodecSupport, ChangedSchemaMergeTest, LukeRequestHandlerTest, TestManagedSchemaFieldResource, CursorMarkTest, TestRequestStatusCollectionAPI, PreAnalyzedFieldTest, TestReversedWildcardFilterFactory, TestHighFrequencyDictionaryFactory, BasicDistributedZk2Test, OpenCloseCoreStressTest, ZkControllerTest, TestNonDefinedSimilarityFactory, SuggesterWFSTTest, TestRealTimeGet, TestConfig, DirectSolrSpellCheckerTest, SOLR749Test, TestOmitPositions, TestDocumentBuilder, XmlUpdateRequestHandlerTest, FunctionTest, LeaderElectionTest, TestRTGBase, ReturnFieldsTest, LeaderElectionIntegrationTest, TestSchemaVersionResource, HdfsLockFactoryTest, CSVRequestHandlerTest, TestBinaryField, QueryParsingTest, QueryElevationComponentTest, RequiredFieldsTest, DistributedQueryComponentCustomSortTest, FieldMutatingUpdateProcessorTest, TestPHPSerializedResponseWriter, TestComponentsName, TestMergePolicyConfig, AbstractAnalyticsStatsTest, HdfsWriteToMultipleCollectionsTest, DistanceFunctionTest, TestReplicationHandler, TestCSVLoader, TestCollationFieldDocValues, TestPartialUpdateDeduplication, SoftAutoCommitTest, TestDistributedGrouping, TestSolrJ, BasicFunctionalityTest, TestConfigSets, TestSolrXmlPersistor, TestLMDirichletSimilarityFactory, TestUpdate, PolyFieldTest, FastVectorHighlighterTest, TestSimpleQParserPlugin, TestJmxMonitoredMap, TestTrie, TestValueSourceCache, ZkCLITest, DistributedExpandComponentTest, SpatialFilterTest, TestCollapseQParserPlugin, TestCustomSort, DeleteReplicaTest, SearchHandlerTest, RequestHandlersTest, TestIndexSearcher, DistributedQueryElevationComponentTest, NoCacheHeaderTest, DocValuesMultiTest, TestRandomDVFaceting, HdfsBasicDistributedZkTest, MultiTermTest, OutputWriterTest, DocValuesMissingTest, BadCopyFieldTest, TestQuerySenderListener, TestCoreContainer, ChaosMonkeyNothingIsSafeTest, PrimitiveFieldTypeTest, RegexBoostProcessorTest, TestWriterPerf, TestSort, AbstractAnalyticsFacetTest, NumericFieldsTest, SolrIndexConfigTest, TestSolrQueryParserResource, TestArbitraryIndexDir, TestSolrXmlPersistence, TestAddFieldRealTimeGet, NotRequiredUniqueKeyTest, TestCollectionAPI, OpenExchangeRatesOrgProviderTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestStressUserVersions, RecoveryZkTest, TestCollationField, TestZkChroot, ModifyConfFileTest, ClusterStateUpdateTest, TestPseudoReturnFields, TestSolrCoreProperties, SolrIndexSplitterTest, TestDistributedSearch, SystemInfoHandlerTest, TestFieldTypeResource, DistribCursorPagingTest, DirectSolrConnectionTest, TestFiltering, UpdateParamsTest, QueryFacetTest, StatsComponentTest, TestQueryUtils, RemoteQueryErrorTest, TestCoreDiscovery, TestStressReorder, UnloadDistributedZkTest, SyncSliceTest, OverseerTest, BasicZkTest, FullSolrCloudDistribCmdsTest, ZkSolrClientTest, ShardRoutingCustomTest, SliceStateUpdateTest, TestFaceting, TestHashPartitioner, DistributedSpellCheckComponentTest, TestJoin, TestReload, TestStressVersions, HardAutoCommitTest, DistributedTermsComponentTest, TestRangeQuery, SimpleFacetsTest, SolrCoreTest, SpellCheckComponentTest, TestGroupingSearch, SolrCmdDistributorTest, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTest, TestFunctionQuery, TestLazyCores, TestBadConfig, OverseerCollectionProcessorTest, CurrencyFieldOpenExchangeTest, CurrencyFieldXmlFileTest, SimplePostToolTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, TestFoldingMultitermQuery, DocValuesTest, SuggesterTSTTest, SuggesterTest, TestStressLucene, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, TestRemoteStreaming, TestSolrDeletionPolicy1, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, DebugComponentTest, CacheHeaderTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, StandardRequestHandlerTest, DocumentAnalysisRequestHandlerTest, TestQueryTypes, FileBasedSpellCheckerTest, TermsComponentTest, TermVectorComponentTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, SolrPluginUtilsTest, IndexSchemaTest, JSONWriterTest, TestCSVResponseWriter, MBeansHandlerTest, UniqFieldsUpdateProcessorFactoryTest, JsonLoaderTest, PingRequestHandlerTest, TestBinaryResponseWriter, HighlighterConfigTest, TestQuerySenderNoQuery, CopyFieldTest, ResponseLogComponentTest, BadComponentTest, TestStressRecovery, TestSolrDeletionPolicy2, SampleTest, TestDocSet, MinimalSchemaTest, ExternalFileFieldSortTest, TestPhraseSuggestions, TestCharFilters, SynonymTokenizerTest, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TimeZoneUtilsTest, ResourceLoaderTest, ScriptEngineTest, URLClassifyProcessorTest, PluginInfoTest, TestFastLRUCache, DateMathParserTest, TestSolrXMLSerializer, TestSystemIdResolver, PrimUtilsTest, DOMUtilTest, TestLRUCache, TestUtils, ZkNodePropsTest, UUIDFieldTest, FileUtilsTest, CursorPagingTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestHighlightDedupGrouping, NoFacetTest, ExpressionTest, FieldFacetExtrasTest, AliasIntegrationTest, AssignTest, AsyncMigrateRouteKeyTest, CollectionsAPIAsyncDistributedZkTest, ConnectionManagerTest, DeleteInactiveReplicaTest, DeleteShardTest, DistribDocExpirationUpdateProcessorTest, OverseerRolesTest, OverseerStatusTest]
[junit4] Completed in 213.10s, 1 test, 1 error <<< FAILURES!
[...truncated 212 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1296: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:920: There were test failures: 398 suites, 1669 tests, 1 error, 1 failure, 42 ignored (15 assumptions)
Total time: 94 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure