You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/19 11:36:12 UTC

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1345 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1345/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 17260 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-map-reduce/test/temp/junit4-J0-20140219_102051_529.syserr
   [junit4] >>> JVM J0: stderr (verbatim) ----
   [junit4] 2014-02-19 10:21:08.337 java[246:4f0f] Unable to load realm info from SCDynamicStore
   [junit4] <<< JVM J0: EOF ----

[...truncated 26336 lines...]
changes-to-html:
    [mkdir] Created dir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/docs/changes
      [get] Getting: https://issues.apache.org/jira/rest/api/2/project/LUCENE
      [get] To: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/docs/changes/jiraVersionList.json
     [exec] Bareword found where operator expected at (eval 1) line 3, near "<title>System"
     [exec] 	(Missing operator before System?)
     [exec] ERROR eval'ing munged JSON string ||<html>
     [exec]   <head>
     [exec]     <title>System Maintenance</title>
     [exec]   </head>
     [exec]   <body>
     [exec]     <h1>Maintenance in progress</h1>
     [exec]     <p>This system is currently down for maintenance. More details may be
     [exec]     available from the <a href='http://monitoring.apache.org/status/'>
     [exec]     ASF Public Network Status</a> page.</p>
     [exec]   </body>
     [exec] </html>
     [exec] ||: Can't find string terminator "'" anywhere before EOF at (eval 1) line 8.
     [exec] 

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:57: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build.xml:208: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build.xml:543: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:2323: exec returned: 255

Total time: 113 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1348 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1348/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 10069 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20140220_015750_197.syserr
   [junit4] >>> JVM J0: stderr (verbatim) ----
   [junit4] java(215,0x15078d000) malloc: *** error for object 0x1d077bec0: pointer being freed was not allocated
   [junit4] *** set a breakpoint in malloc_error_break to debug
   [junit4] <<< JVM J0: EOF ----

[...truncated 1 lines...]
   [junit4] ERROR: JVM J0 ended with an exception, command line: /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/jre/bin/java -XX:-UseCompressedOops -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/heapdumps -Dtests.prefix=tests -Dtests.seed=DCF3BCBF79E36A4E -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=5.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/tools/junit4/logging.properties -Dtests.nightly=false -Dtests.weekly=false -Dtests.slow=true -Dtests.asserts.gracious=false -Dtests.multiplier=1 -DtempDir=. -Djava.io.tmpdir=. -Djunit4.tempDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp -Dclover.db.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/clover/db -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Djava.security.policy=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/tools/junit4/tests.policy -Dlucene.version=5.0-SNAPSHOT -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Dtests.disableHdfs=true -Dfile.encoding=US-ASCII -classpath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/classes/test:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-test-framework/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/test-framework/lib/junit4-ant-2.0.13.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/test-framework/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/codecs/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-solrj/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/analysis/common/lucene-analyzers-common-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/codecs/lucene-codecs-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/highlighter/lucene-highlighter-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/memory/lucene-memory-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/misc/lucene-misc-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/spatial/lucene-spatial-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/expressions/lucene-expressions-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/suggest/lucene-suggest-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/grouping/lucene-grouping-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/queries/lucene-queries-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/queryparser/lucene-queryparser-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/join/lucene-join-5.0-SNAPSHOT.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/antlr-runtime-3.5.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/asm-4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/asm-commons-4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-cli-1.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-codec-1.7.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-configuration-1.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-fileupload-1.2.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/commons-lang-2.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/concurrentlinkedhashmap-lru-1.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/dom4j-1.6.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/guava-14.0.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-annotations-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-auth-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-common-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hadoop-hdfs-2.2.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/hppc-0.5.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/joda-time-2.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/org.restlet-2.1.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/org.restlet.ext.servlet-2.1.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/protobuf-java-2.5.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/lib/spatial4j-0.4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/commons-io-2.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/httpclient-4.3.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/httpcore-4.3.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/httpmime-4.3.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/jcl-over-slf4j-1.6.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/jul-to-slf4j-1.6.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/log4j-1.2.16.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/noggit-0.5.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/slf4j-api-1.6.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/slf4j-log4j12-1.6.6.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/wstx-asl-3.2.7.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/solrj/lib/zookeeper-3.4.5.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-continuation-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-deploy-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-http-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-io-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-jmx-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-security-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-server-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-servlet-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-util-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-webapp-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/jetty-xml-8.1.10.v20130312.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/lib/servlet-api-3.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/build/core/classes/java:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/test-framework/lib/junit-4.10.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/test-framework/lib/randomizedtesting-runner-2.0.13.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/antlr-runtime-3.5.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/asm-4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/asm-commons-4.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/cglib-nodep-2.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/commons-collections-3.2.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/dom4j-1.6.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/easymock-3.0.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/hadoop-common-2.2.0-tests.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/hadoop-hdfs-2.2.0-tests.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/hppc-0.5.2.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/javax.servlet-api-3.0.1.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/jersey-core-1.8.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/jetty-6.1.26.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/jetty-util-6.1.26.jar:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/test-lib/objenesis-1.2.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-launcher.jar:/Users/jenkins/.ant/lib/ivy-2.3.0.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-antlr.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bcel.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-bsf.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-log4j.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-oro.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-regexp.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-resolver.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-apache-xalan2.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-logging.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-commons-net.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jai.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-javamail.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jdepend.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jmf.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-jsch.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-junit4.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-netrexx.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-swing.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant-testutil.jar:/Users/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2/lib/ant.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home/lib/tools.jar:/Users/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.0.13.jar -ea:org.apache.lucene... -ea:org.apache.solr... com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -flush -eventsfile /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20140220_015750_197.events @/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/temp/junit4-J0-20140220_015750_197.suites
   [junit4] ERROR: JVM J0 ended with an exception: Forked process returned with error code: 134. Very likely a JVM crash.  Process output piped in logs above.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1373)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:119)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:911)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:908)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4] 	at java.lang.Thread.run(Thread.java:744)

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:433: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1268: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:901: At least one slave process threw an exception, first: Forked process returned with error code: 134. Very likely a JVM crash.  Process output piped in logs above.

Total time: 35 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1347 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1347/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 16925 lines...]
   [junit4] JVM J0: stderr was not empty, see: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/contrib/solr-map-reduce/test/temp/junit4-J0-20140219_212537_802.syserr
   [junit4] >>> JVM J0: stderr (verbatim) ----
   [junit4] 2014-02-19 21:25:53.991 java[255:5b07] Unable to load realm info from SCDynamicStore
   [junit4] <<< JVM J0: EOF ----

[...truncated 40506 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:392: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:87: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:185: The following files are missing svn:eol-style (or binary svn:mime-type):
* ./solr/licenses/Saxon-HE-9.5.1-4.jar.sha1
* ./solr/licenses/argparse4j-0.4.3.jar.sha1
* ./solr/licenses/jackson-annotations-2.3.1.jar.sha1
* ./solr/licenses/jackson-core-2.3.1.jar.sha1
* ./solr/licenses/jackson-databind-2.3.1.jar.sha1
* ./solr/licenses/kite-morphlines-avro-0.11.0.jar.sha1
* ./solr/licenses/kite-morphlines-core-0.11.0-tests.jar.sha1
* ./solr/licenses/kite-morphlines-core-0.11.0.jar.sha1
* ./solr/licenses/kite-morphlines-hadoop-sequencefile-0.11.0.jar.sha1
* ./solr/licenses/kite-morphlines-json-0.11.0.jar.sha1
* ./solr/licenses/kite-morphlines-saxon-0.11.0.jar.sha1
* ./solr/licenses/kite-morphlines-tika-core-0.11.0.jar.sha1
* ./solr/licenses/kite-morphlines-tika-decompress-0.11.0.jar.sha1
* ./solr/licenses/kite-morphlines-twitter-0.11.0.jar.sha1

Total time: 131 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1346 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1346/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([81ADABC4117F38AF:4B25DC66205893]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:140)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)


REGRESSION:  org.apache.solr.cloud.ShardSplitTest.testDistribSearch

Error Message:
Wrong doc count on shard1_0 expected:<61> but was:<62>

Stack Trace:
java.lang.AssertionError: Wrong doc count on shard1_0 expected:<61> but was:<62>
	at __randomizedtesting.SeedInfo.seed([81ADABC4117F38AF:4B25DC66205893]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.ShardSplitTest.checkDocCountsAndShardStates(ShardSplitTest.java:476)
	at org.apache.solr.cloud.ShardSplitTest.splitByUniqueKeyTest(ShardSplitTest.java:248)
	at org.apache.solr.cloud.ShardSplitTest.doTest(ShardSplitTest.java:112)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
	at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:744)




Build Log:
[...truncated 10210 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> 731032 T2471 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 731032 T2471 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 731039 T2471 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1392821238025
   [junit4]   2> 731040 T2471 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 731041 T2472 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 731141 T2471 oasc.ZkTestServer.run start zk server on port:50019
   [junit4]   2> 731144 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 731150 T2478 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34d19c40 name:ZooKeeperConnection Watcher:127.0.0.1:50019 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 731151 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 731151 T2471 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 731165 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 731169 T2480 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e8a28f9 name:ZooKeeperConnection Watcher:127.0.0.1:50019/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 731170 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 731170 T2471 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 731179 T2471 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 731186 T2471 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 731192 T2471 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 731200 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 731201 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 731215 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 731216 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 731227 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 731228 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 731236 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 731236 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 731243 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 731244 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 731251 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 731252 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 731258 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 731259 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 731266 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 731266 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 731273 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 731274 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 731283 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 731284 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 731294 T2471 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 731295 T2471 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 731311 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 731316 T2482 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f13cd2e name:ZooKeeperConnection Watcher:127.0.0.1:50019/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 731317 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 731666 T2471 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 731673 T2471 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 731680 T2471 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50023
   [junit4]   2> 731688 T2471 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 731689 T2471 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 731690 T2471 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392821238311
   [junit4]   2> 731690 T2471 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392821238311/'
   [junit4]   2> 731725 T2471 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392821238311/solr.xml
   [junit4]   2> 731805 T2471 oasc.CoreContainer.<init> New CoreContainer 1261066725
   [junit4]   2> 731806 T2471 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392821238311/]
   [junit4]   2> 731807 T2471 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 731807 T2471 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 731807 T2471 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 731807 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 731808 T2471 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 731808 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 731808 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 731809 T2471 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 731809 T2471 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 731823 T2471 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 731824 T2471 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 731824 T2471 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 731825 T2471 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50019/solr
   [junit4]   2> 731825 T2471 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 731827 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 731831 T2493 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22a50b05 name:ZooKeeperConnection Watcher:127.0.0.1:50019 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 731833 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 731841 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 731846 T2495 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2663960 name:ZooKeeperConnection Watcher:127.0.0.1:50019/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 731847 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 731851 T2471 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 731861 T2471 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 731872 T2471 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 731877 T2471 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50023_
   [junit4]   2> 731880 T2471 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50023_
   [junit4]   2> 731889 T2471 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 731894 T2471 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 731908 T2471 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50023_
   [junit4]   2> 731909 T2471 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 731915 T2471 oasc.Overseer.start Overseer (id=91279932655665156-127.0.0.1:50023_-n_0000000000) starting
   [junit4]   2> 731923 T2471 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 731933 T2497 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 731934 T2471 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 731936 T2497 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 731940 T2471 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 731945 T2471 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 731953 T2496 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 731959 T2498 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 731959 T2498 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 731962 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731962 T2498 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 731966 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 731968 T2496 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50023",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50023_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 731968 T2496 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 731968 T2496 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 731977 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 731978 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 732965 T2498 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 732965 T2498 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392821238311/collection1
   [junit4]   2> 732965 T2498 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 732968 T2498 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 732968 T2498 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 732972 T2498 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 732972 T2498 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392821238311/collection1/'
   [junit4]   2> 732973 T2498 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392821238311/collection1/lib/classes/' to classloader
   [junit4]   2> 732974 T2498 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392821238311/collection1/lib/README' to classloader
   [junit4]   2> 733037 T2498 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 733100 T2498 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 733103 T2498 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 733112 T2498 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 733596 T2498 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 733599 T2498 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 733601 T2498 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 733608 T2498 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 733667 T2498 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 733668 T2498 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1392821238311/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/control/data/
   [junit4]   2> 733668 T2498 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e501d80
   [junit4]   2> 733669 T2498 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/control/data
   [junit4]   2> 733670 T2498 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/control/data/index/
   [junit4]   2> 733670 T2498 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 733671 T2498 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/control/data/index
   [junit4]   2> 733672 T2498 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.5029296875, noCFSRatio=0.1511814183922737]
   [junit4]   2> 733676 T2498 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 733677 T2498 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 733680 T2498 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 733680 T2498 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 733681 T2498 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 733681 T2498 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 733681 T2498 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 733681 T2498 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 733682 T2498 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 733682 T2498 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 733682 T2498 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 733683 T2498 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 733683 T2498 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 733684 T2498 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 733684 T2498 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 733684 T2498 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 733685 T2498 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 733686 T2498 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 733686 T2498 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 733696 T2498 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 733700 T2498 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 733700 T2498 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 733701 T2498 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=68.8828125, floorSegmentMB=1.09375, forceMergeDeletesPctAllowed=19.782073167964345, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 733702 T2498 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 733703 T2498 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 733703 T2498 oass.SolrIndexSearcher.<init> Opening Searcher@9fdecf7[collection1] main
   [junit4]   2> 733709 T2499 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9fdecf7[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 733711 T2498 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 733712 T2471 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 733713 T2502 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50023 collection:control_collection shard:shard1
   [junit4]   2> 733713 T2471 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 733715 T2502 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 733718 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 733722 T2504 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f6bb888 name:ZooKeeperConnection Watcher:127.0.0.1:50019/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 733722 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 733726 T2471 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 733731 T2502 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 733732 T2471 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 733736 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733738 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 733742 T2502 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 733742 T2502 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2893 name=collection1 org.apache.solr.core.SolrCore@23a4540a url=https://127.0.0.1:50023/collection1 node=127.0.0.1:50023_ C2893_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:50023, core=collection1, node_name=127.0.0.1:50023_}
   [junit4]   2> 733742 T2502 C2893 P50023 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50023/collection1/
   [junit4]   2> 733743 T2502 C2893 P50023 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 733743 T2502 C2893 P50023 oasc.SyncStrategy.syncToMe https://127.0.0.1:50023/collection1/ has no replicas
   [junit4]   2> 733744 T2502 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50023/collection1/ shard1
   [junit4]   2> 733744 T2502 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 733747 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733751 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 733752 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 733769 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733773 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 733780 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733889 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 733889 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 733925 T2502 oasc.ZkController.register We are https://127.0.0.1:50023/collection1/ and leader is https://127.0.0.1:50023/collection1/
   [junit4]   2> 733925 T2502 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50023
   [junit4]   2> 733925 T2502 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 733926 T2502 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 733926 T2502 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 733929 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733929 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733929 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 733931 T2502 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 733936 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 733937 T2496 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50023",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50023_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 733945 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 734052 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 734052 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 734109 T2471 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 734110 T2471 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 734118 T2471 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 734119 T2471 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50027
   [junit4]   2> 734122 T2471 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 734122 T2471 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 734122 T2471 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392821240718
   [junit4]   2> 734123 T2471 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392821240718/'
   [junit4]   2> 734153 T2471 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392821240718/solr.xml
   [junit4]   2> 734239 T2471 oasc.CoreContainer.<init> New CoreContainer 1988924499
   [junit4]   2> 734240 T2471 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392821240718/]
   [junit4]   2> 734242 T2471 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 734242 T2471 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 734242 T2471 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 734243 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 734243 T2471 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 734243 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 734244 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 734244 T2471 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 734244 T2471 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 734259 T2471 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 734259 T2471 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 734260 T2471 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 734260 T2471 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50019/solr
   [junit4]   2> 734260 T2471 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 734262 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 734266 T2515 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@607661b8 name:ZooKeeperConnection Watcher:127.0.0.1:50019 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 734268 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 734273 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 734277 T2517 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@592bee1c name:ZooKeeperConnection Watcher:127.0.0.1:50019/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 734278 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 734291 T2471 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 735301 T2471 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50027_
   [junit4]   2> 735304 T2471 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50027_
   [junit4]   2> 735312 T2495 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 735312 T2504 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 735313 T2517 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 735329 T2518 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 735329 T2518 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 735332 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 735333 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 735334 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 735333 T2518 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 735339 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 735341 T2496 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50027",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50027_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 735341 T2496 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 735342 T2496 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 735349 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 735455 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 735455 T2517 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 735455 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 736336 T2518 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 736337 T2518 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392821240718/collection1
   [junit4]   2> 736337 T2518 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 736340 T2518 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 736341 T2518 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 736348 T2518 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 736349 T2518 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392821240718/collection1/'
   [junit4]   2> 736351 T2518 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392821240718/collection1/lib/classes/' to classloader
   [junit4]   2> 736352 T2518 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392821240718/collection1/lib/README' to classloader
   [junit4]   2> 736436 T2518 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 736512 T2518 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 736516 T2518 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 736522 T2518 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 737059 T2518 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 737062 T2518 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 737064 T2518 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 737074 T2518 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 737184 T2518 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 737185 T2518 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1392821240718/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty1/
   [junit4]   2> 737185 T2518 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e501d80
   [junit4]   2> 737187 T2518 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty1
   [junit4]   2> 737188 T2518 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty1/index/
   [junit4]   2> 737189 T2518 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 737192 T2518 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty1/index
   [junit4]   2> 737192 T2518 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.5029296875, noCFSRatio=0.1511814183922737]
   [junit4]   2> 737198 T2518 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 737199 T2518 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 737208 T2518 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 737209 T2518 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 737209 T2518 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 737209 T2518 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 737210 T2518 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 737210 T2518 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 737210 T2518 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 737211 T2518 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 737211 T2518 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 737211 T2518 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 737212 T2518 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 737212 T2518 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 737212 T2518 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 737213 T2518 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 737213 T2518 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 737214 T2518 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 737215 T2518 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 737224 T2518 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 737227 T2518 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 737228 T2518 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 737228 T2518 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=68.8828125, floorSegmentMB=1.09375, forceMergeDeletesPctAllowed=19.782073167964345, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 737230 T2518 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 737230 T2518 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 737230 T2518 oass.SolrIndexSearcher.<init> Opening Searcher@6d993b94[collection1] main
   [junit4]   2> 737234 T2519 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d993b94[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 737236 T2518 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 737238 T2471 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 737238 T2471 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 737238 T2522 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50027 collection:collection1 shard:shard3
   [junit4]   2> 737241 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 737258 T2522 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 737263 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737264 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737264 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737266 T2522 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 737266 T2522 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2894 name=collection1 org.apache.solr.core.SolrCore@10131686 url=https://127.0.0.1:50027/collection1 node=127.0.0.1:50027_ C2894_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50027, core=collection1, node_name=127.0.0.1:50027_}
   [junit4]   2> 737267 T2522 C2894 P50027 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50027/collection1/
   [junit4]   2> 737267 T2522 C2894 P50027 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 737267 T2522 C2894 P50027 oasc.SyncStrategy.syncToMe https://127.0.0.1:50027/collection1/ has no replicas
   [junit4]   2> 737268 T2522 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50027/collection1/ shard3
   [junit4]   2> 737268 T2522 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 737274 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 737283 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737288 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 737289 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 737290 T2517 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 737302 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 737309 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737414 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 737414 T2517 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 737416 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 737440 T2522 oasc.ZkController.register We are https://127.0.0.1:50027/collection1/ and leader is https://127.0.0.1:50027/collection1/
   [junit4]   2> 737440 T2522 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50027
   [junit4]   2> 737440 T2522 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 737441 T2522 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 737441 T2522 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 737444 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737444 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737445 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737447 T2522 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 737450 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 737452 T2496 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50027",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50027_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 737460 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 737569 T2517 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 737570 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 737570 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 737627 T2471 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 737627 T2471 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 737634 T2471 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 737637 T2471 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50030
   [junit4]   2> 737640 T2471 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 737640 T2471 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 737641 T2471 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392821244227
   [junit4]   2> 737641 T2471 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392821244227/'
   [junit4]   2> 737670 T2471 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392821244227/solr.xml
   [junit4]   2> 737755 T2471 oasc.CoreContainer.<init> New CoreContainer 212682302
   [junit4]   2> 737756 T2471 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392821244227/]
   [junit4]   2> 737758 T2471 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 737758 T2471 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 737758 T2471 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 737759 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 737760 T2471 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 737760 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 737760 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 737761 T2471 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 737761 T2471 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 737776 T2471 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 737776 T2471 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 737776 T2471 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 737777 T2471 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50019/solr
   [junit4]   2> 737777 T2471 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 737779 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 737783 T2533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3dd2f74e name:ZooKeeperConnection Watcher:127.0.0.1:50019 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 737784 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 737789 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 737793 T2535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e12f4c1 name:ZooKeeperConnection Watcher:127.0.0.1:50019/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 737794 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 737808 T2471 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 738818 T2471 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50030_
   [junit4]   2> 738821 T2471 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50030_
   [junit4]   2> 738830 T2535 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 738830 T2504 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 738830 T2495 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 738830 T2517 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 738848 T2536 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 738848 T2536 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 738850 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 738851 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 738851 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 738850 T2536 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 738855 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 738857 T2496 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50030",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50030_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 738857 T2496 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 738858 T2496 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 738863 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 738971 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 738972 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 738972 T2535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 738972 T2517 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 739855 T2536 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 739856 T2536 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392821244227/collection1
   [junit4]   2> 739856 T2536 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 739860 T2536 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 739860 T2536 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 739866 T2536 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 739867 T2536 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392821244227/collection1/'
   [junit4]   2> 739869 T2536 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392821244227/collection1/lib/classes/' to classloader
   [junit4]   2> 739870 T2536 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392821244227/collection1/lib/README' to classloader
   [junit4]   2> 739949 T2536 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 740004 T2536 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 740007 T2536 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 740014 T2536 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 740571 T2536 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 740575 T2536 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 740577 T2536 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 740583 T2536 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 740654 T2536 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 740656 T2536 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1392821244227/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty2/
   [junit4]   2> 740656 T2536 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e501d80
   [junit4]   2> 740657 T2536 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty2
   [junit4]   2> 740657 T2536 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty2/index/
   [junit4]   2> 740658 T2536 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 740659 T2536 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty2/index
   [junit4]   2> 740660 T2536 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=27, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.5029296875, noCFSRatio=0.1511814183922737]
   [junit4]   2> 740663 T2536 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 740663 T2536 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 740667 T2536 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 740667 T2536 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 740668 T2536 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 740668 T2536 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 740668 T2536 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 740669 T2536 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 740669 T2536 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 740670 T2536 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 740670 T2536 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 740670 T2536 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 740671 T2536 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 740671 T2536 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 740672 T2536 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 740673 T2536 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 740673 T2536 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 740674 T2536 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 740674 T2536 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 740685 T2536 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 740689 T2536 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 740689 T2536 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 740690 T2536 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=68.8828125, floorSegmentMB=1.09375, forceMergeDeletesPctAllowed=19.782073167964345, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 740692 T2536 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1392821238025/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 740692 T2536 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 740692 T2536 oass.SolrIndexSearcher.<init> Opening Searcher@1dfc6021[collection1] main
   [junit4]   2> 740696 T2537 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1dfc6021[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 740699 T2536 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 740702 T2471 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 740702 T2471 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 740702 T2540 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50030 collection:collection1 shard:shard2
   [junit4]   2> 740704 T2540 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 740716 T2540 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 740722 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740722 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740723 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740726 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 740727 T2540 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 740727 T2540 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2895 name=collection1 org.apache.solr.core.SolrCore@2b8b76e4 url=https://127.0.0.1:50030/collection1 node=127.0.0.1:50030_ C2895_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50030, core=collection1, node_name=127.0.0.1:50030_}
   [junit4]   2> 740727 T2540 C2895 P50030 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50030/collection1/
   [junit4]   2> 740728 T2540 C2895 P50030 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 740728 T2540 C2895 P50030 oasc.SyncStrategy.syncToMe https://127.0.0.1:50030/collection1/ has no replicas
   [junit4]   2> 740728 T2540 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50030/collection1/ shard2
   [junit4]   2> 740728 T2540 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 740737 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740740 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 740740 T2517 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 740740 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 740744 T2535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 740758 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 740765 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740883 T2517 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 740883 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 740883 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 740884 T2535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 740908 T2540 oasc.ZkController.register We are https://127.0.0.1:50030/collection1/ and leader is https://127.0.0.1:50030/collection1/
   [junit4]   2> 740908 T2540 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50030
   [junit4]   2> 740908 T2540 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 740908 T2540 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 740909 T2540 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 740911 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740911 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740911 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 740914 T2540 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 740915 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 740916 T2496 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50030",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50030_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 740923 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 741029 T2495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 741030 T2535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 741030 T2504 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 741030 T2517 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 741101 T2471 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 741102 T2471 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 741108 T2471 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 741110 T2471 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50033
   [junit4]   2> 741113 T2471 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 741114 T2471 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 741114 T2471 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392821247690
   [junit4]   2> 741115 T2471 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392821247690/'
   [junit4]   2> 741145 T2471 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392821247690/solr.xml
   [junit4]   2> 741236 T2471 oasc.CoreContainer.<init> New CoreContainer 608790109
   [junit4]   2> 741237 T2471 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1392821247690/]
   [junit4]   2> 741238 T2471 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 741238 T2471 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 741238 T2471 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 741239 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 741239 T2471 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 741239 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 741240 T2471 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 741240 T2471 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 741240 T2471 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 741252 T2471 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 741252 T2471 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 741253 T2471 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 741253 T2471 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50019/solr
   [junit4]   2> 741253 T2471 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 741255 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 741259 T2551 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44948cba name:ZooKeeperConnection Watcher:127.0.0.1:50019 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 741260 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 741269 T2471 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 741271 T2553 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50233769 name:ZooKeeperConnection Watcher:127.0.0.1:50019/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 741272 T2471 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 741289 T2471 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 742303 T2471 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50033_
   [junit4]   2> 742314 T2471 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50033_
   [junit4]   2> 742330 T2535 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 742332 T2495 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 742333 T2517 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 742333 T2553 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 742333 T2504 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 742363 T2554 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 742363 T2554 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 742368 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 742368 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 742369 T2495 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 742368 T2554 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 742379 T2496 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 742382 T2496 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50033",
   [junit4]   2> 	  "core":"collec

[...truncated too long message...]

tion$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=hr, timezone=America/Cayenne
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=1,free=79169800,total=337715200
   [junit4]   2> NOTE: All tests run in this JVM: [TestCursorMarkWithoutUniqueKey, TestStressRecovery, OverseerCollectionProcessorTest, DirectUpdateHandlerOptimizeTest, FileUtilsTest, TestFieldTypeCollectionResource, PreAnalyzedUpdateProcessorTest, AnalysisAfterCoreReloadTest, ClusterStateTest, TestSerializedLuceneMatchVersion, TestCollationFieldDocValues, BasicDistributedZk2Test, TestLuceneMatchVersion, TriLevelCompositeIdRoutingTest, TestPhraseSuggestions, TestNRTOpen, TestCustomSort, LukeRequestHandlerTest, WordBreakSolrSpellCheckerTest, LoggingHandlerTest, UpdateRequestProcessorFactoryTest, TestSurroundQueryParser, TestCollapseQParserPlugin, SliceStateUpdateTest, FunctionTest, RemoteQueryErrorTest, TestSuggestSpellingConverter, SystemInfoHandlerTest, TestNonNRTOpen, LeaderElectionIntegrationTest, SuggesterFSTTest, SOLR749Test, EnumFieldTest, TestSystemIdResolver, MoreLikeThisHandlerTest, OutputWriterTest, PluginInfoTest, HdfsCollectionsAPIDistributedZkTest, TestNonDefinedSimilarityFactory, CopyFieldTest, TestComponentsName, TestGroupingSearch, TestStandardQParsers, TestUpdate, AddSchemaFieldsUpdateProcessorFactoryTest, DateFieldTest, TestRandomMergePolicy, SortByFunctionTest, TestQueryTypes, TestFoldingMultitermQuery, TestSolrQueryParserResource, CircularListTest, SpatialFilterTest, SolrRequestParserTest, CustomCollectionTest, AlternateDirectoryTest, TermVectorComponentDistributedTest, SolrXmlInZkTest, DocumentBuilderTest, FieldAnalysisRequestHandlerTest, MBeansHandlerTest, RecoveryZkTest, TestFastOutputStream, ScriptEngineTest, SpellCheckCollatorTest, DirectUpdateHandlerTest, TestDocBasedVersionConstraints, LeaderElectionTest, RangeFacetTest, TestFaceting, DistributedQueryElevationComponentTest, CacheHeaderTest, RegexBoostProcessorTest, TermVectorComponentTest, SolrPluginUtilsTest, BinaryUpdateRequestHandlerTest, IndexSchemaRuntimeFieldTest, ChaosMonkeySafeLeaderTest, HdfsChaosMonkeySafeLeaderTest, TestReplicationHandler, TestInfoStreamLogging, TestDistributedSearch, TestCoreContainer, PingRequestHandlerTest, TestAnalyzeInfixSuggestions, TestNumberUtils, OverseerRolesTest, BasicFunctionalityTest, TestRangeQuery, TestOverriddenPrefixQueryForCustomFieldType, CursorPagingTest, TestSolrXml, TestModifyConfFiles, CurrencyFieldOpenExchangeTest, TestBinaryResponseWriter, TestRecovery, DeleteShardTest, TestSearchPerf, RequiredFieldsTest, OverseerTest, DistanceFunctionTest, TestDistributedGrouping, HighlighterConfigTest, TestMergePolicyConfig, TimeZoneUtilsTest, TestStressLucene, XmlUpdateRequestHandlerTest, SimpleFacetsTest, SchemaVersionSpecificBehaviorTest, TestRandomDVFaceting, JSONWriterTest, TestFileDictionaryLookup, SuggesterTSTTest, TestInitQParser, TestLMDirichletSimilarityFactory, TestDefaultSearchFieldResource, TestPerFieldSimilarity, SolrCoreCheckLockOnStartupTest, PreAnalyzedFieldTest, StatelessScriptUpdateProcessorFactoryTest, ShowFileRequestHandlerTest, ChaosMonkeyNothingIsSafeTest, SearchHandlerTest, TestXIncludeConfig, TestCloudManagedSchemaAddField, QueryEqualityTest, DistributedDebugComponentTest, TestSolrXmlPersistence, BasicDistributedZkTest, PathHierarchyTokenizerFactoryTest, TestCloudManagedSchema, FileBasedSpellCheckerTest, SolrIndexConfigTest, TestDistributedMissingSort, SyncSliceTest, TestCollationField, TestSolr4Spatial, TestReversedWildcardFilterFactory, OpenCloseCoreStressTest, QueryFacetTest, ConnectionManagerTest, QueryParsingTest, TestStressReorder, TestSolrXMLSerializer, TestMultiCoreConfBootstrap, TestCopyFieldCollectionResource, ShardRoutingTest, ResponseLogComponentTest, RAMDirectoryFactoryTest, TestSort, TestClassNameShortening, TestFieldTypeResource, StatsComponentTest, MultiTermTest, TestSolrCoreProperties, TestDocumentBuilder, TestUniqueKeyFieldResource, DistributedSuggestComponentTest, EchoParamsTest, SliceStateTest, TestRTGBase, FieldMutatingUpdateProcessorTest, TestBlendedInfixSuggestions, TestAtomicUpdateErrorCases, IndexSchemaTest, TestStressVersions, DefaultValueUpdateProcessorTest, OpenExchangeRatesOrgProviderTest, HdfsBasicDistributedZkTest, NoCacheHeaderTest, TestCodecSupport, TestDocSet, FastVectorHighlighterTest, HdfsLockFactoryTest, ZkCLITest, HdfsDirectoryTest, TestReload, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, BasicZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, ZkSolrClientTest, TestZkChroot, ShardRoutingCustomTest, AutoCommitTest, TestHashPartitioner, DistributedSpellCheckComponentTest, ZkControllerTest, TestRealTimeGet, TestJoin, HardAutoCommitTest, DistributedTermsComponentTest, SolrCoreTest, SpellCheckComponentTest, SolrCmdDistributorTest, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTest, TestFiltering, TestFunctionQuery, TestLazyCores, TestBadConfig, TestIndexSearcher, HighlighterTest, SoftAutoCommitTest, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, CoreAdminHandlerTest, DocValuesTest, SuggesterTest, TestTrie, SuggesterWFSTTest, TestCSVLoader, PolyFieldTest, TestPseudoReturnFields, TestWordDelimiterFilterFactory, TestRemoteStreaming, DocValuesMultiTest, TestSolrDeletionPolicy1, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, DebugComponentTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, TestQueryUtils, StandardRequestHandlerTest, TestWriterPerf, DirectSolrSpellCheckerTest, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, TestOmitPositions, TermsComponentTest, RequestHandlersTest, TestValueSourceCache, TestIndexingPerformance, TestSolrQueryParser, TestArbitraryIndexDir, TestJmxIntegration, ReturnFieldsTest, TestCSVResponseWriter, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, JsonLoaderTest, TestPartialUpdateDeduplication, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestLFUCache, TestQuerySenderListener, UpdateParamsTest, TestSolrIndexConfig, TestQuerySenderNoQuery, BadComponentTest, TestSolrDeletionPolicy2, SampleTest, TestBinaryField, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, TestConfig, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, DirectSolrConnectionTest, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestCharFilters, SynonymTokenizerTest, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestDefaultSimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, ResourceLoaderTest, URLClassifyProcessorTest, TestFastLRUCache, DateMathParserTest, PrimUtilsTest, SpellingQueryConverterTest, DOMUtilTest, TestSolrJ, TestLRUCache, TestUtils, ZkNodePropsTest, UUIDFieldTest, SolrTestCaseJ4Test, TestHighlightDedupGrouping, AbstractAnalyticsStatsTest, NoFacetTest, ExpressionTest, AbstractAnalyticsFacetTest, FieldFacetExtrasTest, AliasIntegrationTest, AssignTest, DeleteReplicaTest, DistribCursorPagingTest, MigrateRouteKeyTest, ShardSplitTest]
   [junit4] Completed in 43.25s, 1 test, 1 failure <<< FAILURES!

[...truncated 154 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:433: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1268: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:901: There were test failures: 372 suites, 1604 tests, 2 failures, 40 ignored (6 assumptions)

Total time: 88 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure