You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/08/14 01:13:46 UTC

[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build # 6999 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6999/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

24 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1008, name=HashSessionScavenger-16, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1008, name=HashSessionScavenger-16, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.lang.Throwable.fillInStackTrace(Native Method)
	at java.lang.Throwable.<init>(Throwable.java:67)
	at java.lang.Throwable.<init>(Throwable.java:78)
	at java.lang.Error.<init>(Error.java:81)
	at java.lang.VirtualMachineError.<init>(VirtualMachineError.java:63)
	at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:69)
	at java.security.AccessController.getProtectionDomains(Native Method)
	at java.security.AccessController.checkPermission(AccessController.java:88)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
	at java.lang.Thread.setContextClassLoader(Thread.java:797)
	at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:308)
	at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
	at java.util.Timer$TimerImpl.run(Timer.java:296)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=994, name=qtp220461705-994, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=994, name=qtp220461705-994, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.HashMap$1.iterator(HashMap.java:672)
	at java.util.HashSet.iterator(HashSet.java:173)
	at sun.nio.ch.Util$2.iterator(Util.java:315)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=966, name=SyncThread:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=966, name=SyncThread:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:165)
	at org.apache.log4j.Category.forcedLog(Category.java:391)
	at org.apache.log4j.Category.log(Category.java:856)
	at org.slf4j.impl.Log4jLoggerAdapter.error(Log4jLoggerAdapter.java:576)
	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:151)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1026, name=qtp-1726714643-1026, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1026, name=qtp-1726714643-1026, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.HashMap$1.iterator(HashMap.java:672)
	at java.util.HashSet.iterator(HashSet.java:173)
	at sun.nio.ch.Util$2.iterator(Util.java:315)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=992, name=HashSessionScavenger-15, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=992, name=HashSessionScavenger-15, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.lang.Throwable.fillInStackTrace(Native Method)
	at java.lang.Throwable.<init>(Throwable.java:67)
	at java.lang.Throwable.<init>(Throwable.java:78)
	at java.lang.Error.<init>(Error.java:81)
	at java.lang.VirtualMachineError.<init>(VirtualMachineError.java:63)
	at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:69)
	at java.security.AccessController.toArrayOfProtectionDomains(AccessController.java:184)
	at java.security.AccessController.checkPermission(AccessController.java:102)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
	at java.lang.Thread.setContextClassLoader(Thread.java:797)
	at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:308)
	at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
	at java.util.Timer$TimerImpl.run(Timer.java:296)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=993, name=qtp220461705-993, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=993, name=qtp220461705-993, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.lang.Throwable.fillInStackTrace(Native Method)
	at java.lang.Throwable.<init>(Throwable.java:67)
	at java.lang.Throwable.<init>(Throwable.java:78)
	at java.lang.Error.<init>(Error.java:81)
	at java.lang.VirtualMachineError.<init>(VirtualMachineError.java:63)
	at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:69)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:708)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1029, name=qtp-1726714643-1029, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1029, name=qtp-1726714643-1029, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.concurrent.ConcurrentHashMap$KeySet.iterator(ConcurrentHashMap.java:1273)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1.run(SelectorManager.java:712)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1042, name=qtp1556593299-1042, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1042, name=qtp1556593299-1042, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.addWaiter(AbstractQueuedSynchronizer.java:617)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1208)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:225)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:301)
	at org.eclipse.jetty.util.BlockingArrayQueue.offer(BlockingArrayQueue.java:224)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.dispatch(QueuedThreadPool.java:359)
	at org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager.dispatch(SelectChannelConnector.java:300)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:708)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1048, name=qtp1556593299-1048, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1048, name=qtp1556593299-1048, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1210, name=qtp220461705-1210, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1210, name=qtp220461705-1210, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1109, name=qtp1556593299-1109, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1109, name=qtp1556593299-1109, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.concurrent.ConcurrentHashMap$KeySet.iterator(ConcurrentHashMap.java:1273)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1.run(SelectorManager.java:712)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1106, name=qtp220461705-1106, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1106, name=qtp220461705-1106, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1043, name=qtp1556593299-1043, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1043, name=qtp1556593299-1043, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.lang.Throwable.fillInStackTrace(Native Method)
	at java.lang.Throwable.<init>(Throwable.java:67)
	at java.lang.Throwable.<init>(Throwable.java:78)
	at java.lang.Error.<init>(Error.java:81)
	at java.lang.VirtualMachineError.<init>(VirtualMachineError.java:63)
	at java.lang.OutOfMemoryError.<init>(OutOfMemoryError.java:69)
	at java.util.HashMap$1.iterator(HashMap.java:672)
	at java.util.HashSet.iterator(HashSet.java:173)
	at sun.nio.ch.Util$2.iterator(Util.java:315)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1024, name=HashSessionScavenger-17, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1024, name=HashSessionScavenger-17, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.security.AccessController.toArrayOfProtectionDomains(AccessController.java:184)
	at java.security.AccessController.checkPermission(AccessController.java:102)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
	at java.lang.Thread.setContextClassLoader(Thread.java:797)
	at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:344)
	at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
	at java.util.Timer$TimerImpl.run(Timer.java:296)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=964, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=964, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.ArrayList$SubList.listIterator(ArrayList.java:1024)
	at java.util.AbstractList.listIterator(AbstractList.java:629)
	at java.util.ArrayList$SubList.iterator(ArrayList.java:1016)
	at java.util.AbstractCollection.toArray(AbstractCollection.java:363)
	at java.util.regex.Pattern.split(Pattern.java:1238)
	at java.util.regex.Pattern.split(Pattern.java:1270)
	at java.lang.String.split(String.java:1749)
	at org.apache.zookeeper.jmx.MBeanRegistry.tokenize(MBeanRegistry.java:182)
	at org.apache.zookeeper.jmx.MBeanRegistry.makeObjectName(MBeanRegistry.java:204)
	at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:115)
	at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:132)
	at org.apache.zookeeper.server.ServerCnxnFactory.unregisterConnection(ServerCnxnFactory.java:137)
	at org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:972)
	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:352)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1000, name=qtp220461705-1000, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1000, name=qtp220461705-1000, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.concurrent.ConcurrentHashMap$KeySet.iterator(ConcurrentHashMap.java:1273)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1.run(SelectorManager.java:712)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1096, name=qtp1639013060-1096, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1096, name=qtp1639013060-1096, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1009, name=qtp1639013060-1009 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1009, name=qtp1639013060-1009 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:304)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1025, name=qtp-1726714643-1025, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1025, name=qtp-1726714643-1025, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.util.HashMap$1.iterator(HashMap.java:672)
	at java.util.HashSet.iterator(HashSet.java:173)
	at sun.nio.ch.Util$2.iterator(Util.java:315)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1209, name=qtp1556593299-1209, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1209, name=qtp1556593299-1209, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1041, name=HashSessionScavenger-18, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1041, name=HashSessionScavenger-18, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
	at java.security.AccessController.getProtectionDomains(Native Method)
	at java.security.AccessController.checkPermission(AccessController.java:88)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
	at java.lang.Thread.setContextClassLoader(Thread.java:797)
	at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:344)
	at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
	at java.util.Timer$TimerImpl.run(Timer.java:296)


FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
Server at http://127.0.0.1:38771/n_b/m/onenodecollectioncore returned non ok status:404, message:Can not find: /n_b/m/onenodecollectioncore/update

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:38771/n_b/m/onenodecollectioncore returned non ok status:404, message:Can not find: /n_b/m/onenodecollectioncore/update
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976:88DA56E28154494A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:88)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:613)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:780)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)




Build Log:
[...truncated 9663 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
   [junit4]   2> 1279686 T3618 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /n_b/m
   [junit4]   2> 1279690 T3618 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1376429262138
   [junit4]   2> 1279691 T3618 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 1279693 T3619 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1279793 T3618 oasc.ZkTestServer.run start zk server on port:56032
   [junit4]   2> 1279795 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1280075 T3625 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3afff252 name:ZooKeeperConnection Watcher:127.0.0.1:56032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1280075 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1280076 T3618 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1280083 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1280084 T3627 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ef341f0e name:ZooKeeperConnection Watcher:127.0.0.1:56032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1280085 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1280085 T3618 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1280088 T3618 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1280090 T3618 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1280092 T3618 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1280094 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1280095 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1280098 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1280099 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1280101 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1280102 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1280104 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1280105 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1280107 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1280108 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1280110 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1280110 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1280112 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1280113 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1280115 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1280115 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1280118 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1280118 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1280120 T3618 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1280121 T3618 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1280251 T3618 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1280293 T3618 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56324
   [junit4]   2> 1280294 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1280297 T3618 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1280298 T3618 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571
   [junit4]   2> 1280299 T3618 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/'
   [junit4]   2> 1280317 T3618 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/solr.xml
   [junit4]   2> 1280360 T3618 oasc.CoreContainer.<init> New CoreContainer 2074598383
   [junit4]   2> 1280361 T3618 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/]
   [junit4]   2> 1280363 T3618 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1280364 T3618 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1280365 T3618 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1280366 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1280367 T3618 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1280368 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1280368 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1280369 T3618 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1280370 T3618 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1280371 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 1280382 T3618 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56032/solr
   [junit4]   2> 1280383 T3618 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1280385 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1280387 T3638 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72ebed10 name:ZooKeeperConnection Watcher:127.0.0.1:56032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1280387 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1280389 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 1280397 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1280399 T3640 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76ca6e5b name:ZooKeeperConnection Watcher:127.0.0.1:56032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1280399 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1280401 T3618 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1280404 T3618 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1280408 T3618 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1280410 T3618 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56324_n_b%2Fm
   [junit4]   2> 1280411 T3618 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56324_n_b%2Fm
   [junit4]   2> 1280414 T3618 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1280418 T3618 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1280419 T3618 oasc.Overseer.start Overseer (id=90205668124721155-127.0.0.1:56324_n_b%2Fm-n_0000000000) starting
   [junit4]   2> 1280422 T3618 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1280425 T3642 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1280426 T3618 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1280429 T3618 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1280431 T3618 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1280434 T3641 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1280437 T3643 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1280437 T3643 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1280438 T3643 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1281944 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1281945 T3641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56324/n_b/m",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:56324_n_b%2Fm",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1281946 T3641 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1281946 T3641 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1281952 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1282442 T3643 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1282442 T3643 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/collection1
   [junit4]   2> 1282443 T3643 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1282444 T3643 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1282444 T3643 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1282445 T3643 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/collection1/'
   [junit4]   2> 1282447 T3643 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/collection1/lib/README' to classloader
   [junit4]   2> 1282447 T3643 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/collection1/lib/classes/' to classloader
   [junit4]   2> 1282505 T3643 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1282556 T3643 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1282558 T3643 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1282571 T3643 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1283040 T3643 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1283046 T3643 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1283050 T3643 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1283103 T3643 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1283108 T3643 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1283114 T3643 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1283118 T3643 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1283119 T3643 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1283119 T3643 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1283129 T3643 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1283129 T3643 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1283130 T3643 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1283131 T3643 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1376429262571/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/control/data/
   [junit4]   2> 1283131 T3643 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f99c27e
   [junit4]   2> 1283132 T3643 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/control/data
   [junit4]   2> 1283133 T3643 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/control/data/index/
   [junit4]   2> 1283134 T3643 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1283137 T3643 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/control/data/index
   [junit4]   2> 1283139 T3643 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@bad203ce lockFactory=org.apache.lucene.store.NativeFSLockFactory@e554df6d),segFN=segments_1,generation=1}
   [junit4]   2> 1283140 T3643 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1283144 T3643 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1283145 T3643 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1283145 T3643 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1283147 T3643 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1283147 T3643 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1283149 T3643 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1283149 T3643 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1283166 T3643 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1283167 T3643 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1283170 T3643 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1283186 T3643 oass.SolrIndexSearcher.<init> Opening Searcher@5bf3ca55 main
   [junit4]   2> 1283189 T3643 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1283190 T3643 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1283193 T3644 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5bf3ca55 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 1283194 T3643 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1283195 T3643 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56324/n_b/m collection:control_collection shard:shard1
   [junit4]   2> 1283195 T3643 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 1283204 T3643 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1283209 T3643 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1283212 T3643 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1283212 T3643 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1283212 T3643 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56324/n_b/m/collection1/
   [junit4]   2> 1283213 T3643 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1283213 T3643 oasc.SyncStrategy.syncToMe http://127.0.0.1:56324/n_b/m/collection1/ has no replicas
   [junit4]   2> 1283214 T3643 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56324/n_b/m/collection1/ shard1
   [junit4]   2> 1283214 T3643 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1283456 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1283465 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1283520 T3643 oasc.ZkController.register We are http://127.0.0.1:56324/n_b/m/collection1/ and leader is http://127.0.0.1:56324/n_b/m/collection1/
   [junit4]   2> 1283520 T3643 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56324/n_b/m
   [junit4]   2> 1283521 T3643 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1283521 T3643 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1283522 T3643 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1283523 T3643 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1283524 T3618 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1283525 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1283526 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 1283533 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 1283537 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1283538 T3647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55c058e0 name:ZooKeeperConnection Watcher:127.0.0.1:56032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1283538 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1283540 T3618 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1283542 T3618 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 1283648 T3618 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1283665 T3618 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59579
   [junit4]   2> 1283666 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1283667 T3618 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1283668 T3618 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990
   [junit4]   2> 1283668 T3618 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/'
   [junit4]   2> 1283677 T3618 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/solr.xml
   [junit4]   2> 1283746 T3618 oasc.CoreContainer.<init> New CoreContainer 712316351
   [junit4]   2> 1283747 T3618 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/]
   [junit4]   2> 1283749 T3618 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1283750 T3618 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1283751 T3618 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1283752 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1283752 T3618 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1283753 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1283754 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1283755 T3618 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1283756 T3618 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1283757 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 1283765 T3618 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56032/solr
   [junit4]   2> 1283766 T3618 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1283769 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1283770 T3658 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e737e00 name:ZooKeeperConnection Watcher:127.0.0.1:56032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1283771 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1283782 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 1283792 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1283816 T3660 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ae959547 name:ZooKeeperConnection Watcher:127.0.0.1:56032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1283817 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1283821 T3618 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1284824 T3618 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59579_n_b%2Fm
   [junit4]   2> 1284827 T3618 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59579_n_b%2Fm
   [junit4]   2> 1284830 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1284830 T3647 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1284831 T3660 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1284833 T3640 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 1284842 T3661 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1284842 T3661 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1284846 T3661 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1284988 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1284989 T3641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:56324/n_b/m",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:56324_n_b%2Fm",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1284995 T3641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59579/n_b/m",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:59579_n_b%2Fm",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1284996 T3641 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1284996 T3641 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1285000 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1285000 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1285000 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1285846 T3661 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1285847 T3661 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/collection1
   [junit4]   2> 1285848 T3661 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1285849 T3661 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1285849 T3661 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1285850 T3661 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/collection1/'
   [junit4]   2> 1285852 T3661 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/collection1/lib/README' to classloader
   [junit4]   2> 1285852 T3661 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/collection1/lib/classes/' to classloader
   [junit4]   2> 1285893 T3661 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1285941 T3661 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1285945 T3661 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1285954 T3661 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1286433 T3661 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1286439 T3661 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1286442 T3661 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1286453 T3661 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1286457 T3661 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1286461 T3661 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1286463 T3661 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1286463 T3661 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1286463 T3661 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1286464 T3661 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1286465 T3661 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1286465 T3661 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1286465 T3661 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1376429265990/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty1/
   [junit4]   2> 1286466 T3661 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f99c27e
   [junit4]   2> 1286467 T3661 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty1
   [junit4]   2> 1286467 T3661 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty1/index/
   [junit4]   2> 1286467 T3661 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1286468 T3661 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty1/index
   [junit4]   2> 1286469 T3661 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@ebdce726 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56c9cf76),segFN=segments_1,generation=1}
   [junit4]   2> 1286470 T3661 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1286472 T3661 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1286472 T3661 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1286475 T3661 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1286476 T3661 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1286477 T3661 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1286478 T3661 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1286478 T3661 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1286479 T3661 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1286480 T3661 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1286482 T3661 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1286486 T3661 oass.SolrIndexSearcher.<init> Opening Searcher@13a1840f main
   [junit4]   2> 1286489 T3661 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1286489 T3661 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1286499 T3661 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1286499 T3661 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59579/n_b/m collection:collection1 shard:shard1
   [junit4]   2> 1286500 T3661 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 1286502 T3662 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13a1840f main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 1286522 T3661 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1286528 T3661 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1286531 T3661 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1286532 T3661 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1286532 T3661 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59579/n_b/m/collection1/
   [junit4]   2> 1286533 T3661 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1286533 T3661 oasc.SyncStrategy.syncToMe http://127.0.0.1:59579/n_b/m/collection1/ has no replicas
   [junit4]   2> 1286534 T3661 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59579/n_b/m/collection1/ shard1
   [junit4]   2> 1286534 T3661 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1288013 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1288025 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1288025 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1288029 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1288056 T3661 oasc.ZkController.register We are http://127.0.0.1:59579/n_b/m/collection1/ and leader is http://127.0.0.1:59579/n_b/m/collection1/
   [junit4]   2> 1288056 T3661 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59579/n_b/m
   [junit4]   2> 1288057 T3661 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1288057 T3661 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1288057 T3661 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1288059 T3661 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1288060 T3618 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1288061 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1288061 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 1288159 T3618 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1288166 T3618 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38771
   [junit4]   2> 1288167 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1288168 T3618 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1288169 T3618 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515
   [junit4]   2> 1288170 T3618 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/'
   [junit4]   2> 1288195 T3618 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/solr.xml
   [junit4]   2> 1288243 T3618 oasc.CoreContainer.<init> New CoreContainer -648773389
   [junit4]   2> 1288244 T3618 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/]
   [junit4]   2> 1288246 T3618 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1288247 T3618 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 1288248 T3618 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1288248 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1288249 T3618 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1288250 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1288252 T3618 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1288253 T3618 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1288254 T3618 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1288255 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 1288274 T3618 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:56032/solr
   [junit4]   2> 1288275 T3618 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1288281 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1288283 T3678 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@90e2423d name:ZooKeeperConnection Watcher:127.0.0.1:56032 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1288283 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1288284 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 1288291 T3618 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1288300 T3680 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@758e0c99 name:ZooKeeperConnection Watcher:127.0.0.1:56032/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1288301 T3618 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1288306 T3618 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1289310 T3618 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38771_n_b%2Fm
   [junit4]   2> 1289312 T3618 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38771_n_b%2Fm
   [junit4]   2> 1289315 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 1289315 T3647 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1289316 T3680 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1289316 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1289316 T3660 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1289317 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1289317 T3640 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 1289342 T3681 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 1289342 T3681 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1289344 T3681 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1289539 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1289540 T3641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:59579/n_b/m",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "node_name":"127.0.0.1:59579_n_b%2Fm",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1289545 T3641 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38771/n_b/m",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "node_name":"127.0.0.1:38771_n_b%2Fm",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 1289546 T3641 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1289546 T3641 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1289550 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1289550 T3680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1289550 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1289550 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1290344 T3681 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1290345 T3681 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/collection1
   [junit4]   2> 1290345 T3681 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1290346 T3681 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1290347 T3681 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1290348 T3681 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/collection1/'
   [junit4]   2> 1290349 T3681 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/collection1/lib/README' to classloader
   [junit4]   2> 1290350 T3681 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/collection1/lib/classes/' to classloader
   [junit4]   2> 1290399 T3681 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 1290449 T3681 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1290451 T3681 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1290460 T3681 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1290959 T3681 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1290965 T3681 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1290967 T3681 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1290978 T3681 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1290984 T3681 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1290995 T3681 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1291001 T3681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1291001 T3681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1291003 T3681 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1291006 T3681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1291007 T3681 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1291008 T3681 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1291009 T3681 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1376429270515/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty2/
   [junit4]   2> 1291010 T3681 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f99c27e
   [junit4]   2> 1291015 T3681 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty2
   [junit4]   2> 1291015 T3681 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty2/index/
   [junit4]   2> 1291016 T3681 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1291017 T3681 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1376429262138/jetty2/index
   [junit4]   2> 1291021 T3681 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@a8d77bfa lockFactory=org.apache.lucene.store.NativeFSLockFactory@c4f0017),segFN=segments_1,generation=1}
   [junit4]   2> 1291021 T3681 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1291033 T3681 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1291033 T3681 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1291035 T3681 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1291036 T3681 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1291037 T3681 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1291038 T3681 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1291039 T3681 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1291040 T3681 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1291040 T3681 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1291042 T3681 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1291046 T3681 oass.SolrIndexSearcher.<init> Opening Searcher@3afcff90 main
   [junit4]   2> 1291049 T3681 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1291049 T3681 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1291055 T3682 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3afcff90 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 1291057 T3681 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1291057 T3681 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38771/n_b/m collection:collection1 shard:shard2
   [junit4]   2> 1291058 T3681 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 1291065 T3681 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1291073 T3681 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1291076 T3681 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1291076 T3681 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1291076 T3681 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38771/n_b/m/collection1/
   [junit4]   2> 1291077 T3681 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1291077 T3681 oasc.SyncStrategy.syncToMe http://127.0.0.1:38771/n_b/m/collection1/ has no replicas
   [junit4]   2> 1291078 T3681 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38771/n_b/m/collection1/ shard2
   [junit4]   2> 1291078 T3681 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1292558 T3641 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1292564 T3680 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1292564 T3660 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1292564 T3640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1292564 T3647 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 1292607 T3681 oasc.ZkController.register We are http://127.0.0.1:38771/n_b/m/collection1/ and leader is http://127.0.0.1:38771/n_b/m/collection1/
   [junit4]   2> 1292607 T3681 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38771/n_b/m
   [junit4]   2> 1292607 T3681 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1292608 T3681 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 1292608 T3681 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1292610 T3681 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 1292611 T3618 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1292612 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1292613 T3618 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 1292723 T3618 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1292729 T3618 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57465
   [junit4]   2> 1292731 T3618 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1292732 T3618 oasc.SolrResourceLoader.locateSolrHome JNDI 

[...truncated too long message...]

un(QueuedThreadPool.java:543)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)Throwable #15: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1024, name=HashSessionScavenger-17, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
   [junit4]    > 	at java.security.AccessController.toArrayOfProtectionDomains(AccessController.java:184)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:102)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
   [junit4]    > 	at java.lang.Thread.setContextClassLoader(Thread.java:797)
   [junit4]    > 	at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:344)
   [junit4]    > 	at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
   [junit4]    > 	at java.util.Timer$TimerImpl.run(Timer.java:296)Throwable #16: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=964, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
   [junit4]    > 	at java.util.ArrayList$SubList.listIterator(ArrayList.java:1024)
   [junit4]    > 	at java.util.AbstractList.listIterator(AbstractList.java:629)
   [junit4]    > 	at java.util.ArrayList$SubList.iterator(ArrayList.java:1016)
   [junit4]    > 	at java.util.AbstractCollection.toArray(AbstractCollection.java:363)
   [junit4]    > 	at java.util.regex.Pattern.split(Pattern.java:1238)
   [junit4]    > 	at java.util.regex.Pattern.split(Pattern.java:1270)
   [junit4]    > 	at java.lang.String.split(String.java:1749)
   [junit4]    > 	at org.apache.zookeeper.jmx.MBeanRegistry.tokenize(MBeanRegistry.java:182)
   [junit4]    > 	at org.apache.zookeeper.jmx.MBeanRegistry.makeObjectName(MBeanRegistry.java:204)
   [junit4]    > 	at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:115)
   [junit4]    > 	at org.apache.zookeeper.jmx.MBeanRegistry.unregister(MBeanRegistry.java:132)
   [junit4]    > 	at org.apache.zookeeper.server.ServerCnxnFactory.unregisterConnection(ServerCnxnFactory.java:137)
   [junit4]    > 	at org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:972)
   [junit4]    > 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:352)
   [junit4]    > 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)Throwable #17: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1000, name=qtp220461705-1000, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
   [junit4]    > 	at java.util.concurrent.ConcurrentHashMap$KeySet.iterator(ConcurrentHashMap.java:1273)
   [junit4]    > 	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet$1.run(SelectorManager.java:712)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)Throwable #18: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1096, name=qtp1639013060-1096, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
   [junit4]    > 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
   [junit4]    > 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
   [junit4]    > 	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)Throwable #19: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1009, name=qtp1639013060-1009 Selector0, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
   [junit4]    > 	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:304)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)Throwable #20: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1025, name=qtp-1726714643-1025, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
   [junit4]    > 	at java.util.HashMap$1.iterator(HashMap.java:672)
   [junit4]    > 	at java.util.HashSet.iterator(HashSet.java:173)
   [junit4]    > 	at sun.nio.ch.Util$2.iterator(Util.java:315)
   [junit4]    > 	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:600)
   [junit4]    > 	at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)Throwable #21: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1209, name=qtp1556593299-1209, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1868)
   [junit4]    > 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2084)
   [junit4]    > 	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:780)Throwable #22: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1041, name=HashSessionScavenger-18, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([93CD8FAF60B2976]:0)
   [junit4]    > 	at java.security.AccessController.getProtectionDomains(Native Method)
   [junit4]    > 	at java.security.AccessController.checkPermission(AccessController.java:88)
   [junit4]    > 	at java.lang.SecurityManager.checkPermission(SecurityManager.java:560)
   [junit4]    > 	at java.lang.Thread.setContextClassLoader(Thread.java:797)
   [junit4]    > 	at org.eclipse.jetty.server.session.HashSessionManager.scavenge(HashSessionManager.java:344)
   [junit4]    > 	at org.eclipse.jetty.server.session.HashSessionManager$2.run(HashSessionManager.java:284)
   [junit4]    > 	at java.util.Timer$TimerImpl.run(Timer.java:296)
   [junit4] Completed on J1 in 7226.21s, 1 test, 23 errors <<< FAILURES!

[...truncated 229 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:389: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:382: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:450: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1230: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:873: There were test failures: 320 suites, 1360 tests, 22 suite-level errors, 2 errors, 215 ignored (5 assumptions)

Total time: 151 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure