You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2012/10/29 08:31:21 UTC

[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 818 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/818/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestBinaryField

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.schema.TestBinaryField:     1) Thread[id=226, name=qtp39737982-226 Acceptor2 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=224, name=qtp39737982-224 Acceptor0 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=225, name=qtp39737982-225 Acceptor1 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=227, name=qtp39737982-227 Acceptor3 SelectChannelConnector@0.0.0.0:21558, state=RUNNABLE, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.schema.TestBinaryField: 
   1) Thread[id=226, name=qtp39737982-226 Acceptor2 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=224, name=qtp39737982-224 Acceptor0 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=225, name=qtp39737982-225 Acceptor1 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=227, name=qtp39737982-227 Acceptor3 SelectChannelConnector@0.0.0.0:21558, state=RUNNABLE, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([E4D6C2186BC87AC4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestBinaryField

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=226, name=qtp39737982-226 Acceptor2 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=224, name=qtp39737982-224 Acceptor0 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=225, name=qtp39737982-225 Acceptor1 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=227, name=qtp39737982-227 Acceptor3 SelectChannelConnector@0.0.0.0:21558, state=RUNNABLE, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=226, name=qtp39737982-226 Acceptor2 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=224, name=qtp39737982-224 Acceptor0 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=225, name=qtp39737982-225 Acceptor1 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=227, name=qtp39737982-227 Acceptor3 SelectChannelConnector@0.0.0.0:21558, state=RUNNABLE, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([E4D6C2186BC87AC4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.NoCacheHeaderTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest:     1) Thread[id=370, name=qtp1021579782-370 Acceptor3 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=368, name=qtp1021579782-368 Acceptor1 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=369, name=qtp1021579782-369 Acceptor2 SelectChannelConnector@0.0.0.0:15218, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=367, name=qtp1021579782-367 Acceptor0 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest: 
   1) Thread[id=370, name=qtp1021579782-370 Acceptor3 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=368, name=qtp1021579782-368 Acceptor1 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=369, name=qtp1021579782-369 Acceptor2 SelectChannelConnector@0.0.0.0:15218, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=367, name=qtp1021579782-367 Acceptor0 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([E4D6C2186BC87AC4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.NoCacheHeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=370, name=qtp1021579782-370 Acceptor3 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=368, name=qtp1021579782-368 Acceptor1 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=369, name=qtp1021579782-369 Acceptor2 SelectChannelConnector@0.0.0.0:15218, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=367, name=qtp1021579782-367 Acceptor0 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=370, name=qtp1021579782-370 Acceptor3 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=368, name=qtp1021579782-368 Acceptor1 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=369, name=qtp1021579782-369 Acceptor2 SelectChannelConnector@0.0.0.0:15218, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=367, name=qtp1021579782-367 Acceptor0 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([E4D6C2186BC87AC4]:0)




Build Log:
[...truncated 7785 lines...]
[junit4:junit4] Suite: org.apache.solr.schema.TestBinaryField
[junit4:junit4]   2> 1711 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/collection1/'
[junit4:junit4]   2> 2162 T217 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2447 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:21558
[junit4:junit4]   2> 2474 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2474 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example
[junit4:junit4]   2> 2474 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/'
[junit4:junit4]   2> 2505 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2506 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2506 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example
[junit4:junit4]   2> 2507 T217 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/solr.xml
[junit4:junit4]   2> 2507 T217 oasc.CoreContainer.<init> New CoreContainer 427840304
[junit4:junit4]   2> 2508 T217 oasc.CoreContainer$Initializer.initialize no solr.xml file found - using default
[junit4:junit4]   2> 2509 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/'
[junit4:junit4]   2> 2509 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/'
[junit4:junit4]   2> 2533 T217 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2561 T217 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/collection1
[junit4:junit4]   2> 2562 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/collection1/'
[junit4:junit4]   2> 2608 T217 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2645 T217 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2646 T217 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2649 T217 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2715 T217 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2716 T217 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/collection1/data/
[junit4:junit4]   2> 2716 T217 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 2717 T217 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/collection1/data/index/
[junit4:junit4]   2> 2718 T217 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/collection1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2719 T217 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351495733628/example/collection1/data/index forceNew:false
[junit4:junit4]   2> 2720 T217 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7824edc2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3678c533; maxCacheMB=0.4541015625 maxMergeSizeMB=0.232421875)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2721 T217 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2725 T217 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2725 T217 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2725 T217 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2725 T217 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2726 T217 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2737 T217 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2744 T217 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2748 T217 oass.SolrIndexSearcher.<init> Opening Searcher@44d20052 main
[junit4:junit4]   2> 2749 T217 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 2750 T217 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2750 T217 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2751 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2751 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2752 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2752 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2752 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2753 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2753 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2754 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2754 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2755 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2772 T217 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2772 T228 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@44d20052 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2772 T217 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2773 T217 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2773 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2820 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C10 name=collection1 org.apache.solr.core.SolrCore@2f935b72
[junit4:junit4]   2> 3981 T230 C10 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7824edc2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3678c533; maxCacheMB=0.4541015625 maxMergeSizeMB=0.232421875)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3982 T230 C10 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3983 T230 C10 UPDATE [collection1] webapp=/example path=/update params={wt=javabin&version=2} {add=[1]} 0 51
[junit4:junit4]   2> 4104 T231 C10 UPDATE [collection1] webapp=/example path=/update params={wt=javabin&version=2} {add=[2]} 0 1
[junit4:junit4]   2> 4108 T230 C10 UPDATE [collection1] webapp=/example path=/update params={wt=javabin&version=2} {add=[3]} 0 1
[junit4:junit4]   2> 4138 T231 C10 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 4141 T231 C10 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7824edc2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3678c533; maxCacheMB=0.4541015625 maxMergeSizeMB=0.232421875)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7824edc2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3678c533; maxCacheMB=0.4541015625 maxMergeSizeMB=0.232421875)),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Lucene41_0.doc, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0.fdt, _0_Lucene41_0.tip]
[junit4:junit4]   2> 4142 T231 C10 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 4144 T231 C10 oass.SolrIndexSearcher.<init> Opening Searcher@52bf7b40 main
[junit4:junit4]   2> 4144 T231 C10 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4145 T231 C10 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 4145 T228 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@52bf7b40 main{StandardDirectoryReader(segments_2:3 _0(4.1):C3)}
[junit4:junit4]   2> 4145 T231 C10 UPDATE [collection1] webapp=/example path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 8
[junit4:junit4]   2> 4209 T230 C10 REQ [collection1] webapp=/example path=/select params={wt=javabin&q=*:*&version=2} hits=3 status=0 QTime=39 
[junit4:junit4]   2> 4273 T217 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=427840304
[junit4:junit4]   2> 4273 T217 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2f935b72
[junit4:junit4]   2> 4274 T217 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 4274 T217 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 4275 T217 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 4275 T217 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 4276 T217 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 4278 T217 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/example,null}
[junit4:junit4]   2> 4378 T217 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 4382 T216 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 24470 T216 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 4 threads leaked from SUITE scope at org.apache.solr.schema.TestBinaryField: 
[junit4:junit4]   2> 	   1) Thread[id=226, name=qtp39737982-226 Acceptor2 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=224, name=qtp39737982-224 Acceptor0 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=225, name=qtp39737982-225 Acceptor1 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=227, name=qtp39737982-227 Acceptor3 SelectChannelConnector@0.0.0.0:21558, state=RUNNABLE, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 24471 T216 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=226, name=qtp39737982-226 Acceptor2 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	   2) Thread[id=224, name=qtp39737982-224 Acceptor0 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	   3) Thread[id=225, name=qtp39737982-225 Acceptor1 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	   4) Thread[id=227, name=qtp39737982-227 Acceptor3 SelectChannelConnector@0.0.0.0:21558, state=RUNNABLE, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 27515 T216 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=226, name=qtp39737982-226 Acceptor2 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=224, name=qtp39737982-224 Acceptor0 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=225, name=qtp39737982-225 Acceptor1 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=227, name=qtp39737982-227 Acceptor3 SelectChannelConnector@0.0.0.0:21558, state=RUNNABLE, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {id=Lucene41(blocksize=128)}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_EG, timezone=America/Dawson
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=26378752,total=162922496
[junit4:junit4]   2> NOTE: All tests run in this JVM: [StandardRequestHandlerTest, TestHashPartitioner, TestCSVLoader, FieldMutatingUpdateProcessorTest, TestSolrXMLSerializer, TestDocSet, TestFoldingMultitermQuery, SignatureUpdateProcessorFactoryTest, SolrInfoMBeanTest, TestStressVersions, TermVectorComponentTest, TestMultiCoreConfBootstrap, TestQueryUtils, DisMaxRequestHandlerTest, ConvertedLegacyTest, TestBM25SimilarityFactory, UpdateParamsTest, TestBinaryField]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBinaryField -Dtests.seed=E4D6C2186BC87AC4 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_EG -Dtests.timezone=America/Dawson -Dtests.file.encoding=ISO8859-1
[junit4:junit4] ERROR   0.00s J1 | TestBinaryField (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.schema.TestBinaryField: 
[junit4:junit4]    >    1) Thread[id=226, name=qtp39737982-226 Acceptor2 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=224, name=qtp39737982-224 Acceptor0 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=225, name=qtp39737982-225 Acceptor1 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=227, name=qtp39737982-227 Acceptor3 SelectChannelConnector@0.0.0.0:21558, state=RUNNABLE, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([E4D6C2186BC87AC4]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=226, name=qtp39737982-226 Acceptor2 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=224, name=qtp39737982-224 Acceptor0 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=225, name=qtp39737982-225 Acceptor1 SelectChannelConnector@0.0.0.0:21558, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=227, name=qtp39737982-227 Acceptor3 SelectChannelConnector@0.0.0.0:21558, state=RUNNABLE, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([E4D6C2186BC87AC4]:0)
[junit4:junit4] Completed on J1 in 25.98s, 1 test, 2 errors <<< FAILURES!

[...truncated 485 lines...]
[junit4:junit4] Suite: org.apache.solr.servlet.NoCacheHeaderTest
[junit4:junit4]   2> 1 T360 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-NoCacheHeaderTest-1351495846543
[junit4:junit4]   2> 2 T360 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 540 T360 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 829 T360 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:15218
[junit4:junit4]   2> 868 T360 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 868 T360 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 868 T360 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 909 T360 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 909 T360 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 910 T360 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 910 T360 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 910 T360 oasc.CoreContainer.<init> New CoreContainer 105404186
[junit4:junit4]   2> 911 T360 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 911 T360 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 961 T360 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1032 T360 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1033 T360 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1034 T360 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1035 T360 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1081 T360 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1131 T360 oasc.SolrConfig.<init> Loaded SolrConfig: solr/collection1/conf/solrconfig-nocache.xml
[junit4:junit4]   2> 1132 T360 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1144 T360 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1740 T360 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1752 T360 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1756 T360 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1767 T360 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1772 T360 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1776 T360 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1778 T360 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1780 T360 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351495846543/
[junit4:junit4]   2> 1780 T360 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 1781 T360 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351495846543/index/
[junit4:junit4]   2> 1782 T360 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351495846543/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1784 T360 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351495846543/index forceNew:false
[junit4:junit4]   2> 1786 T360 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6537c964 lockFactory=org.apache.lucene.store.NativeFSLockFactory@140da031),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1786 T360 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1787 T360 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1793 T360 oass.SolrIndexSearcher.<init> Opening Searcher@37bc4fcc main
[junit4:junit4]   2> 1794 T360 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1794 T360 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1795 T360 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1795 T360 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1795 T360 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1796 T360 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1796 T360 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1796 T360 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1797 T360 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1797 T360 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1798 T360 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1798 T360 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1798 T360 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1812 T375 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37bc4fcc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1812 T360 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1814 T360 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 1814 T360 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1814 T360 oas.SolrJettyTestBase.createJetty Jetty Assigned Port#15218
[junit4:junit4]   2> 1820 T360 oas.SolrTestCaseJ4.setUp ###Starting testLastModified
[junit4:junit4]   2> 1849 T360 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C31 name=collection1 org.apache.solr.core.SolrCore@5babadb1
[junit4:junit4]   2> 2843 T374 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=12 
[junit4:junit4]   2> 2994 T372 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 2999 T373 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3003 T374 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3006 T372 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3009 T373 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3012 T374 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3015 T372 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3018 T373 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3021 T374 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3022 T360 oas.SolrTestCaseJ4.tearDown ###Ending testLastModified
[junit4:junit4]   2> 3028 T360 oas.SolrTestCaseJ4.setUp ###Starting testCacheControl
[junit4:junit4]   2> 3030 T372 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3033 T373 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3041 T374 C31 REQ [collection1] webapp=/solr path=/select params={q=solr&qt=standard} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3042 T360 oas.SolrTestCaseJ4.tearDown ###Ending testCacheControl
[junit4:junit4]   2> 3049 T360 oas.SolrTestCaseJ4.setUp ###Starting testEtag
[junit4:junit4]   2> 3051 T372 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3055 T373 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3057 T374 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3060 T372 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3063 T373 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3066 T374 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3069 T372 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3072 T373 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3075 T374 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3078 T372 C31 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3079 T360 oas.SolrTestCaseJ4.tearDown ###Ending testEtag
[junit4:junit4]   2> 3125 T360 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=105404186
[junit4:junit4]   2> 3126 T360 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5babadb1
[junit4:junit4]   2> 3126 T360 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 3126 T360 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 3127 T360 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 3127 T360 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 3128 T360 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 3129 T360 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 3230 T360 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 3230 T360 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 3236 T359 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 23351 T359 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest: 
[junit4:junit4]   2> 	   1) Thread[id=370, name=qtp1021579782-370 Acceptor3 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=368, name=qtp1021579782-368 Acceptor1 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=369, name=qtp1021579782-369 Acceptor2 SelectChannelConnector@0.0.0.0:15218, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=367, name=qtp1021579782-367 Acceptor0 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 23351 T359 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=370, name=qtp1021579782-370 Acceptor3 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	   2) Thread[id=368, name=qtp1021579782-368 Acceptor1 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	   3) Thread[id=369, name=qtp1021579782-369 Acceptor2 SelectChannelConnector@0.0.0.0:15218, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	   4) Thread[id=367, name=qtp1021579782-367 Acceptor0 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 26395 T359 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=370, name=qtp1021579782-370 Acceptor3 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=368, name=qtp1021579782-368 Acceptor1 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=369, name=qtp1021579782-369 Acceptor2 SelectChannelConnector@0.0.0.0:15218, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=367, name=qtp1021579782-367 Acceptor0 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=SimpleText, sim=DefaultSimilarity, locale=lt, timezone=Europe/Jersey
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=140288152,total=301727744
[junit4:junit4]   2> NOTE: All tests run in this JVM: [SpellCheckCollatorTest, DistanceFunctionTest, LoggingHandlerTest, TestRTGBase, PreAnalyzedFieldTest, TestFastLRUCache, TestBinaryResponseWriter, HighlighterConfigTest, MoreLikeThisHandlerTest, TestCollationField, UniqFieldsUpdateProcessorFactoryTest, SOLR749Test, CopyFieldTest, SystemInfoHandlerTest, TestFiltering, TestConfig, TestRangeQuery, SolrPluginUtilsTest, TestLuceneMatchVersion, TestRandomFaceting, TestWordDelimiterFilterFactory, TestExtendedDismaxParser, QueryElevationComponentTest, NoCacheHeaderTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=NoCacheHeaderTest -Dtests.seed=E4D6C2186BC87AC4 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=lt -Dtests.timezone=Europe/Jersey -Dtests.file.encoding=ISO8859-1
[junit4:junit4] ERROR   0.00s J0 | NoCacheHeaderTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest: 
[junit4:junit4]    >    1) Thread[id=370, name=qtp1021579782-370 Acceptor3 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=368, name=qtp1021579782-368 Acceptor1 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=369, name=qtp1021579782-369 Acceptor2 SelectChannelConnector@0.0.0.0:15218, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=367, name=qtp1021579782-367 Acceptor0 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([E4D6C2186BC87AC4]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=370, name=qtp1021579782-370 Acceptor3 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=368, name=qtp1021579782-368 Acceptor1 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=369, name=qtp1021579782-369 Acceptor2 SelectChannelConnector@0.0.0.0:15218, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=367, name=qtp1021579782-367 Acceptor0 SelectChannelConnector@0.0.0.0:15218, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([E4D6C2186BC87AC4]:0)
[junit4:junit4] Completed on J0 in 26.57s, 3 tests, 2 errors <<< FAILURES!

[...truncated 107 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1159: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 718 ignored

Total time: 20 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 839 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/839/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.TestDistributedGrouping

Error Message:
36 threads leaked from SUITE scope at org.apache.solr.TestDistributedGrouping:     1) Thread[id=164, name=qtp704420455-164 Acceptor1 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=282, name=qtp860052087-282 Acceptor2 SelectChannelConnector@0.0.0.0:14585, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=150, name=qtp1111180471-150 Acceptor1 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=199, name=qtp748358246-199 Acceptor0 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=266, name=qtp817976996-266 Acceptor0 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=280, name=qtp860052087-280 Acceptor0 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=163, name=qtp704420455-163 Acceptor0 SelectChannelConnector@0.0.0.0:54308, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=166, name=qtp704420455-166 Acceptor3 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=252, name=qtp21028611-252 Acceptor0 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=185, name=qtp1284214182-185 Acceptor0 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=238, name=qtp1918877421-238 Acceptor0 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=239, name=qtp1918877421-239 Acceptor1 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=152, name=qtp1111180471-152 Acceptor3 SelectChannelConnector@0.0.0.0:16172, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=255, name=qtp21028611-255 Acceptor3 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=241, name=qtp1918877421-241 Acceptor3 SelectChannelConnector@0.0.0.0:14582, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=214, name=qtp1819552999-214 Acceptor3 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=149, name=qtp1111180471-149 Acceptor0 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=151, name=qtp1111180471-151 Acceptor2 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=283, name=qtp860052087-283 Acceptor3 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=188, name=qtp1284214182-188 Acceptor3 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=269, name=qtp817976996-269 Acceptor3 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=211, name=qtp1819552999-211 Acceptor0 SelectChannelConnector@0.0.0.0:12020, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=186, name=qtp1284214182-186 Acceptor1 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=165, name=qtp704420455-165 Acceptor2 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=281, name=qtp860052087-281 Acceptor1 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=268, name=qtp817976996-268 Acceptor2 SelectChannelConnector@0.0.0.0:14584, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=187, name=qtp1284214182-187 Acceptor2 SelectChannelConnector@0.0.0.0:31780, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=201, name=qtp748358246-201 Acceptor2 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=200, name=qtp748358246-200 Acceptor1 SelectChannelConnector@0.0.0.0:29763, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=212, name=qtp1819552999-212 Acceptor1 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=202, name=qtp748358246-202 Acceptor3 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=213, name=qtp1819552999-213 Acceptor2 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=240, name=qtp1918877421-240 Acceptor2 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=253, name=qtp21028611-253 Acceptor1 SelectChannelConnector@0.0.0.0:14583, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=267, name=qtp817976996-267 Acceptor1 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=254, name=qtp21028611-254 Acceptor2 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 36 threads leaked from SUITE scope at org.apache.solr.TestDistributedGrouping: 
   1) Thread[id=164, name=qtp704420455-164 Acceptor1 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=282, name=qtp860052087-282 Acceptor2 SelectChannelConnector@0.0.0.0:14585, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=150, name=qtp1111180471-150 Acceptor1 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=199, name=qtp748358246-199 Acceptor0 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=266, name=qtp817976996-266 Acceptor0 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=280, name=qtp860052087-280 Acceptor0 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=163, name=qtp704420455-163 Acceptor0 SelectChannelConnector@0.0.0.0:54308, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=166, name=qtp704420455-166 Acceptor3 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=252, name=qtp21028611-252 Acceptor0 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=185, name=qtp1284214182-185 Acceptor0 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=238, name=qtp1918877421-238 Acceptor0 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=239, name=qtp1918877421-239 Acceptor1 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=152, name=qtp1111180471-152 Acceptor3 SelectChannelConnector@0.0.0.0:16172, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=255, name=qtp21028611-255 Acceptor3 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=241, name=qtp1918877421-241 Acceptor3 SelectChannelConnector@0.0.0.0:14582, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=214, name=qtp1819552999-214 Acceptor3 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=149, name=qtp1111180471-149 Acceptor0 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=151, name=qtp1111180471-151 Acceptor2 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=283, name=qtp860052087-283 Acceptor3 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=188, name=qtp1284214182-188 Acceptor3 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=269, name=qtp817976996-269 Acceptor3 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=211, name=qtp1819552999-211 Acceptor0 SelectChannelConnector@0.0.0.0:12020, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=186, name=qtp1284214182-186 Acceptor1 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=165, name=qtp704420455-165 Acceptor2 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=281, name=qtp860052087-281 Acceptor1 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=268, name=qtp817976996-268 Acceptor2 SelectChannelConnector@0.0.0.0:14584, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=187, name=qtp1284214182-187 Acceptor2 SelectChannelConnector@0.0.0.0:31780, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=201, name=qtp748358246-201 Acceptor2 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=200, name=qtp748358246-200 Acceptor1 SelectChannelConnector@0.0.0.0:29763, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=212, name=qtp1819552999-212 Acceptor1 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=202, name=qtp748358246-202 Acceptor3 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=213, name=qtp1819552999-213 Acceptor2 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=240, name=qtp1918877421-240 Acceptor2 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=253, name=qtp21028611-253 Acceptor1 SelectChannelConnector@0.0.0.0:14583, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=267, name=qtp817976996-267 Acceptor1 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=254, name=qtp21028611-254 Acceptor2 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([8C944CC3AC69D41E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.TestDistributedGrouping

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=164, name=qtp704420455-164 Acceptor1 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=282, name=qtp860052087-282 Acceptor2 SelectChannelConnector@0.0.0.0:14585, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=150, name=qtp1111180471-150 Acceptor1 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=199, name=qtp748358246-199 Acceptor0 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=266, name=qtp817976996-266 Acceptor0 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=280, name=qtp860052087-280 Acceptor0 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=163, name=qtp704420455-163 Acceptor0 SelectChannelConnector@0.0.0.0:54308, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=166, name=qtp704420455-166 Acceptor3 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=252, name=qtp21028611-252 Acceptor0 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=185, name=qtp1284214182-185 Acceptor0 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=238, name=qtp1918877421-238 Acceptor0 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=239, name=qtp1918877421-239 Acceptor1 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=152, name=qtp1111180471-152 Acceptor3 SelectChannelConnector@0.0.0.0:16172, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=255, name=qtp21028611-255 Acceptor3 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=214, name=qtp1819552999-214 Acceptor3 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=241, name=qtp1918877421-241 Acceptor3 SelectChannelConnector@0.0.0.0:14582, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=149, name=qtp1111180471-149 Acceptor0 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=151, name=qtp1111180471-151 Acceptor2 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=283, name=qtp860052087-283 Acceptor3 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=188, name=qtp1284214182-188 Acceptor3 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=186, name=qtp1284214182-186 Acceptor1 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=211, name=qtp1819552999-211 Acceptor0 SelectChannelConnector@0.0.0.0:12020, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=269, name=qtp817976996-269 Acceptor3 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=281, name=qtp860052087-281 Acceptor1 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=165, name=qtp704420455-165 Acceptor2 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=268, name=qtp817976996-268 Acceptor2 SelectChannelConnector@0.0.0.0:14584, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=187, name=qtp1284214182-187 Acceptor2 SelectChannelConnector@0.0.0.0:31780, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=201, name=qtp748358246-201 Acceptor2 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=200, name=qtp748358246-200 Acceptor1 SelectChannelConnector@0.0.0.0:29763, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=212, name=qtp1819552999-212 Acceptor1 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=202, name=qtp748358246-202 Acceptor3 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=213, name=qtp1819552999-213 Acceptor2 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=240, name=qtp1918877421-240 Acceptor2 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=253, name=qtp21028611-253 Acceptor1 SelectChannelConnector@0.0.0.0:14583, state=RUNNABLE, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=267, name=qtp817976996-267 Acceptor1 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=254, name=qtp21028611-254 Acceptor2 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=164, name=qtp704420455-164 Acceptor1 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=282, name=qtp860052087-282 Acceptor2 SelectChannelConnector@0.0.0.0:14585, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=150, name=qtp1111180471-150 Acceptor1 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=199, name=qtp748358246-199 Acceptor0 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=266, name=qtp817976996-266 Acceptor0 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=280, name=qtp860052087-280 Acceptor0 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=163, name=qtp704420455-163 Acceptor0 SelectChannelConnector@0.0.0.0:54308, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=166, name=qtp704420455-166 Acceptor3 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=252, name=qtp21028611-252 Acceptor0 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=185, name=qtp1284214182-185 Acceptor0 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=238, name=qtp1918877421-238 Acceptor0 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=239, name=qtp1918877421-239 Acceptor1 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=152, name=qtp1111180471-152 Acceptor3 SelectChannelConnector@0.0.0.0:16172, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=255, name=qtp21028611-255 Acceptor3 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=214, name=qtp1819552999-214 Acceptor3 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=241, name=qtp1918877421-241 Acceptor3 SelectChannelConnector@0.0.0.0:14582, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=149, name=qtp1111180471-149 Acceptor0 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=151, name=qtp1111180471-151 Acceptor2 SelectChannelConnector@0.0.0.0:16172, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=283, name=qtp860052087-283 Acceptor3 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=188, name=qtp1284214182-188 Acceptor3 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=186, name=qtp1284214182-186 Acceptor1 SelectChannelConnector@0.0.0.0:31780, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=211, name=qtp1819552999-211 Acceptor0 SelectChannelConnector@0.0.0.0:12020, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=269, name=qtp817976996-269 Acceptor3 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=281, name=qtp860052087-281 Acceptor1 SelectChannelConnector@0.0.0.0:14585, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=165, name=qtp704420455-165 Acceptor2 SelectChannelConnector@0.0.0.0:54308, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=268, name=qtp817976996-268 Acceptor2 SelectChannelConnector@0.0.0.0:14584, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=187, name=qtp1284214182-187 Acceptor2 SelectChannelConnector@0.0.0.0:31780, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=201, name=qtp748358246-201 Acceptor2 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=200, name=qtp748358246-200 Acceptor1 SelectChannelConnector@0.0.0.0:29763, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=212, name=qtp1819552999-212 Acceptor1 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=202, name=qtp748358246-202 Acceptor3 SelectChannelConnector@0.0.0.0:29763, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=213, name=qtp1819552999-213 Acceptor2 SelectChannelConnector@0.0.0.0:12020, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=240, name=qtp1918877421-240 Acceptor2 SelectChannelConnector@0.0.0.0:14582, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=253, name=qtp21028611-253 Acceptor1 SelectChannelConnector@0.0.0.0:14583, state=RUNNABLE, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=267, name=qtp817976996-267 Acceptor1 SelectChannelConnector@0.0.0.0:14584, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=254, name=qtp21028611-254 Acceptor2 SelectChannelConnector@0.0.0.0:14583, state=BLOCKED, group=TGRP-TestDistributedGrouping]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([8C944CC3AC69D41E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=253, name=qtp604251701-253 Acceptor2 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=235, name=qtp1558648238-235 Acceptor2 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=201, name=qtp1894659009-201 Acceptor0 SelectChannelConnector@0.0.0.0:47890, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=252, name=qtp604251701-252 Acceptor1 SelectChannelConnector@0.0.0.0:12486, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=186, name=qtp2042731979-186 Acceptor3 SelectChannelConnector@0.0.0.0:22639, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=251, name=qtp604251701-251 Acceptor0 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=184, name=qtp2042731979-184 Acceptor1 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=203, name=qtp1894659009-203 Acceptor2 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=202, name=qtp1894659009-202 Acceptor1 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=236, name=qtp1558648238-236 Acceptor3 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=183, name=qtp2042731979-183 Acceptor0 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=185, name=qtp2042731979-185 Acceptor2 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=204, name=qtp1894659009-204 Acceptor3 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=233, name=qtp1558648238-233 Acceptor0 SelectChannelConnector@0.0.0.0:21320, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=234, name=qtp1558648238-234 Acceptor1 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=254, name=qtp604251701-254 Acceptor3 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=253, name=qtp604251701-253 Acceptor2 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=235, name=qtp1558648238-235 Acceptor2 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=201, name=qtp1894659009-201 Acceptor0 SelectChannelConnector@0.0.0.0:47890, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=252, name=qtp604251701-252 Acceptor1 SelectChannelConnector@0.0.0.0:12486, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=186, name=qtp2042731979-186 Acceptor3 SelectChannelConnector@0.0.0.0:22639, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=251, name=qtp604251701-251 Acceptor0 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=184, name=qtp2042731979-184 Acceptor1 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=203, name=qtp1894659009-203 Acceptor2 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=202, name=qtp1894659009-202 Acceptor1 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=236, name=qtp1558648238-236 Acceptor3 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=183, name=qtp2042731979-183 Acceptor0 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=185, name=qtp2042731979-185 Acceptor2 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=204, name=qtp1894659009-204 Acceptor3 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=233, name=qtp1558648238-233 Acceptor0 SelectChannelConnector@0.0.0.0:21320, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=234, name=qtp1558648238-234 Acceptor1 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=254, name=qtp604251701-254 Acceptor3 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([8C944CC3AC69D41E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=235, name=qtp1558648238-235 Acceptor2 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=253, name=qtp604251701-253 Acceptor2 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=201, name=qtp1894659009-201 Acceptor0 SelectChannelConnector@0.0.0.0:47890, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=252, name=qtp604251701-252 Acceptor1 SelectChannelConnector@0.0.0.0:12486, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=186, name=qtp2042731979-186 Acceptor3 SelectChannelConnector@0.0.0.0:22639, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=251, name=qtp604251701-251 Acceptor0 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=184, name=qtp2042731979-184 Acceptor1 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=203, name=qtp1894659009-203 Acceptor2 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=202, name=qtp1894659009-202 Acceptor1 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=236, name=qtp1558648238-236 Acceptor3 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=183, name=qtp2042731979-183 Acceptor0 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=185, name=qtp2042731979-185 Acceptor2 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=233, name=qtp1558648238-233 Acceptor0 SelectChannelConnector@0.0.0.0:21320, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=204, name=qtp1894659009-204 Acceptor3 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=234, name=qtp1558648238-234 Acceptor1 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=254, name=qtp604251701-254 Acceptor3 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=235, name=qtp1558648238-235 Acceptor2 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=253, name=qtp604251701-253 Acceptor2 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=201, name=qtp1894659009-201 Acceptor0 SelectChannelConnector@0.0.0.0:47890, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=252, name=qtp604251701-252 Acceptor1 SelectChannelConnector@0.0.0.0:12486, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=186, name=qtp2042731979-186 Acceptor3 SelectChannelConnector@0.0.0.0:22639, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=251, name=qtp604251701-251 Acceptor0 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=184, name=qtp2042731979-184 Acceptor1 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=203, name=qtp1894659009-203 Acceptor2 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=202, name=qtp1894659009-202 Acceptor1 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=236, name=qtp1558648238-236 Acceptor3 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=183, name=qtp2042731979-183 Acceptor0 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=185, name=qtp2042731979-185 Acceptor2 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=233, name=qtp1558648238-233 Acceptor0 SelectChannelConnector@0.0.0.0:21320, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=204, name=qtp1894659009-204 Acceptor3 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=234, name=qtp1558648238-234 Acceptor1 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=254, name=qtp604251701-254 Acceptor3 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([8C944CC3AC69D41E]:0)




Build Log:
[...truncated 7773 lines...]
[junit4:junit4] Suite: org.apache.solr.TestDistributedGrouping
[junit4:junit4]   2> 4 T142 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 443 T142 oejs.Server.doStart jetty-8.1.7.v20120910
[junit4:junit4]   2> 716 T142 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:16172
[junit4:junit4]   2> 754 T142 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 755 T142 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 755 T142 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 788 T142 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 789 T142 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 789 T142 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 790 T142 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 790 T142 oasc.CoreContainer.<init> New CoreContainer 1967326322
[junit4:junit4]   2> 791 T142 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 791 T142 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 826 T142 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 869 T142 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 870 T142 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 871 T142 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 872 T142 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 995 T142 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1133 T142 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1134 T142 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1167 T142 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1649 T142 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1661 T142 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1664 T142 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1677 T142 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1681 T142 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1686 T142 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1688 T142 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1690 T142 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.TestDistributedGrouping-1352423571683/control/data/
[junit4:junit4]   2> 1691 T142 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2180b22e
[junit4:junit4]   2> 1692 T142 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.TestDistributedGrouping-1352423571683/control/data/index/
[junit4:junit4]   2> 1693 T142 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.TestDistributedGrouping-1352423571683/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1694 T142 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.TestDistributedGrouping-1352423571683/control/data/index forceNew:false
[junit4:junit4]   2> 1697 T142 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@62469173 lockFactory=org.apache.lucene.store.NativeFSLockFactory@547248ad),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1697 T142 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1698 T142 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 1700 T142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1700 T142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 1701 T142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1702 T142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 1702 T142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1703 T142 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1703 T142 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1704 T142 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1704 T142 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1705 T142 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 1706 T142 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 1706 T142 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1707 T142 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 1707 T142 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 1708 T142 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 1708 T142 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1709 T142 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1709 T142 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1710 T142 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1710 T142 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1711 T142 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1712 T142 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1712 T142 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1713 T142 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 1713 T142 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 1718 T142 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1722 T142 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 1724 T142 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, TRIVIAL.UPDATEPROCESSOR1.JS, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-MINIMAL.XML, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, SOLRCONFIG-MERGEPOLICY.XML, CURRENCY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-IB.XML, SCHEMA-CHARFILTERS.XML, SOLRCONFIG-QUERYSENDER.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, DA_UTF8.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, SCHEMA-REVERSED.XML, STOPTYPES-2.TXT, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SOLRCONFIG-PHRASESUGGEST.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, SCHEMA-SPATIAL.XML, OPEN-EXCHANGE-RATES.JSON, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA_CODEC.XML, SCHEMA.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, BAD-MP-SOLRCONFIG.XML, FRENCHARTICLES.TXT, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, SOLRCONFIG_PERF.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 1728 T142 oass.SolrIndexSearcher.<init> Opening Searcher@74a47bf9 main
[junit4:junit4]   2> 1729 T142 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1729 T142 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1730 T142 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1730 T142 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 1762 T142 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 1810 T142 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1811 T142 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1811 T142 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1811 T142 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1812 T142 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1812 T142 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1813 T142 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1813 T142 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1814 T142 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1814 T142 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1834 T157 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@74a47bf9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1840 T142 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1840 T142 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 1840 T142 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1888 T142 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1906 T142 oejs.Server.doStart jetty-8.1.7.v20120910
[junit4:junit4]   2> 1909 T142 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:54308
[junit4:junit4]   2> 1909 T142 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1910 T142 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1910 T142 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1935 T142 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1935 T142 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1936 T142 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1936 T142 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1936 T142 oasc.CoreContainer.<init> New CoreContainer 341201131
[junit4:junit4]   2> 1937 T142 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1937 T142 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1956 T142 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1970 T142 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1970 T142 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1971 T142 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1972 T142 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2015 T142 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2074 T142 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2075 T142 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2095 T142 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2542 T142 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2553 T142 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2557 T142 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2568 T142 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2572 T142 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2575 T142 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2577 T142 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2578 T142 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.TestDistributedGrouping-1352423571683/shard0/data/
[junit4:junit4]   2> 2579 T142 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2180b22e
[junit4:junit4]   2> 2580 T142 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.TestDistributedGrouping-1352423571683/shard0/data/index/
[junit4:junit4]   2> 2581 T142 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.TestDistributedGrouping-1352423571683/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2581 T142 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.TestDistributedGrouping-1352423571683/shard0/data/index forceNew:false
[junit4:junit4]   2> 2584 T142 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6a636064 lockFactory=org.apache.lucene.store.NativeFSLockFactory@cfae153),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2584 T142 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2585 T142 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 2587 T142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2588 T142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 2588 T142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2589 T142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 2590 T142 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2590 T142 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2591 T142 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2591 T142 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2592 T142 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2593 T142 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 2594 T142 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 2594 T142 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2595 T142 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2595 T142 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2596 T142 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2596 T142 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2597 T142 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2598 T142 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2599 T142 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2599 T142 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2600 T142 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2601 T142 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2601 T142 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2602 T142 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 2603 T142 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 2608 T142 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2611 T142 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 2612 T142 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, TRIVIAL.UPDATEPROCESSOR1.JS, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-MINIMAL.XML, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, SOLRCONFIG-MERGEPOLICY.XML, CURRENCY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-IB.XML, SCHEMA-CHARFILTERS.XML, SOLRCONFIG-QUERYSENDER.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, DA_UTF8.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, SCHEMA-REVERSED.XML, STOPTYPES-2.TXT, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SOLRCONFIG-PHRASESUGGEST.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, SCHEMA-SPATIAL.XML, OPEN-EXCHANGE-RATES.JSON, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA_CODEC.XML, SCHEMA.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, BAD-MP-SOLRCONFIG.XML, FRENCHARTICLES.TXT, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, SOLRCONFIG_PERF.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 2615 T142 oass.SolrIndexSearcher.<init> Opening Searcher@79e4f9af main
[junit4:junit4]   2> 2615 T142 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 2616 T142 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2616 T142 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2617 T142 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 2624 T142 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2662 T142 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2663 T142 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2663 T142 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2663 T142 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2664 T142 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2664 T142 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2664 T142 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2665 T142 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2665 T142 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2665 T142 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2683 T169 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79e4f9af main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2690 T142 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2691 T142 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2691 T142 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2692 T142 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@18430830
[junit4:junit4]   2> 3915 T153 C2 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 3918 T153 C2 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@62469173 lockFactory=org.apache.lucene.store.NativeFSLockFactory@547248ad),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3919 T153 C2 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3919 T153 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 13
[junit4:junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@58d872f5
[junit4:junit4]   2> 4064 T170 C3 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 4067 T170 C3 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6a636064 lockFactory=org.apache.lucene.store.NativeFSLockFactory@cfae153),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4067 T170 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4068 T170 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 4
[junit4:junit4]   2> 4096 T156 C2 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 4097 T156 C2 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@62469173 lockFactory=org.apache.lucene.store.NativeFSLockFactory@547248ad),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@62469173 lockFactory=org.apache.lucene.store.NativeFSLockFactory@547248ad),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 4097 T156 C2 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 4098 T156 C2 oass.SolrIndexSearcher.<init> Opening Searcher@6645f1ca main
[junit4:junit4]   2> 4098 T156 C2 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4099 T156 C2 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 4101 T157 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6645f1ca main{StandardDirectoryReader(segments_2:2)}
[junit4:junit4]   2> 4102 T156 C2 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 6
[junit4:junit4]   2> 4106 T167 C3 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 4107 T167 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6a636064 lockFactory=org.apache.lucene.store.NativeFSLockFactory@cfae153),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6a636064 lockFactory=org.apache.lucene.store.NativeFSLockFactory@cfae153),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 4107 T167 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 4108 T167 C3 oass.SolrIndexSearcher.<init> Opening Searcher@174709c6 main
[junit4:junit4]   2> 4108 T167 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4108 T167 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 4111 T169 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@174709c6 main{StandardDirectoryReader(segments_2:2)}
[junit4:junit4]   2> 4111 T167 C3 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 5
[junit4:junit4]   2> 4321 T155 C2 REQ [collection1] webapp=/solr path=/select params={fl=id,a_si&sort=a_si+asc,+id+asc&q=*:*&group.limit=10&group.field=a_si&distrib=false&group=true&wt=javabin&rows=100&version=2} hits=0 status=0 QTime=186 
[junit4:junit4]   2> 4656 T170 C3 REQ [collection1] webapp=/solr path=/select params={group.distributed.first=true&sort=a_si+asc,+id+asc&group.limit=10&distrib=false&wt=javabin&rows=100&version=2&NOW=1352423576012&shard.url=[ff01::114]:33332/solr|127.0.0.1:54308/solr|[ff01::213]:33332/solr&fl=id&start=0&q=*:*&group.field=a_si&group=true&isShard=true} status=0 QTime=60 
[junit4:junit4]   2> 4710 T167 C3 REQ [collection1] webapp=/solr path=/select params={fl=id,a_si&sort=a_si+asc,+id+asc&shards=[ff01::114]:33332/solr|127.0.0.1:54308/solr|[ff01::213]:33332/solr&q=*:*&group.limit=10&group.field=a_si&group=true&wt=javabin&rows=100&version=2} status=0 QTime=377 
[junit4:junit4]   2> 4765 T156 C2 REQ [collection1] webapp=/solr path=/select params={sort=a_si+asc,+id+asc&group.limit=10&distrib=false&hl.fl=a_t&wt=javabin&hl=true&rows=100&version=2&fl=id,a_si&q=*:*&group.field=a_si&group=true} hits=0 status=0 QTime=52 
[junit4:junit4]   2> 4772 T170 C3 REQ [collection1] webapp=/solr path=/select params={sort=a_si+asc,+id+asc&group.distributed.first=true&group.limit=10&distrib=false&hl.fl=a_t&wt=javabin&hl=false&rows=100&version=2&fl=id&shard.url=[ff01::114]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:54308/solr&NOW=1352423576448&start=0&q=*:*&group.field=a_si&group=true&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 4773 T167 C3 REQ [collection1] webapp=/solr path=/select params={sort=a_si+asc,+id+asc&group.limit=10&hl.fl=a_t&wt=javabin&hl=true&rows=100&version=2&fl=id,a_si&shards=[ff01::114]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:54308/solr&q=*:*&group.field=a_si&group=true} status=0 QTime=4 
[junit4:junit4]   2> 4833 T153 C2 oasr.UnInvertedField.<init> UnInverted multi-valued field {field=a_t,memSize=4288,tindexSize=0,time=0,phase1=0,nTerms=0,bigTerms=0,termInstances=0,uses=0}
[junit4:junit4]   2> 4841 T153 C2 REQ [collection1] webapp=/solr path=/select params={facet=true&sort=a_si+asc,+id+asc&group.limit=10&distrib=false&wt=javabin&rows=100&version=2&fl=id,a_si&q=*:*&group.field=a_si&group=true&facet.field=a_t} hits=0 status=0 QTime=65 
[junit4:junit4]   2> 4855 T167 C3 REQ [collection1] webapp=/solr path=/select params={facet=false&sort=a_si+asc,+id+asc&group.distributed.first=true&group.limit=10&distrib=false&wt=javabin&rows=100&version=2&fl=id&shard.url=[ff01::083]:33332/solr|127.0.0.1:54308/solr&NOW=1352423576528&start=0&q=*:*&group.field=a_si&group=true&facet.field=a_t&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 4890 T167 C3 oasr.UnInvertedField.<init> UnInverted multi-valued field {field=a_t,memSize=4288,tindexSize=0,time=0,phase1=0,nTerms=0,bigTerms=0,termInstances=0,uses=0}
[junit4:junit4]   2> 4891 T167 C3 REQ [collection1] webapp=/solr path=/select params={facet=true&sort=a_si+asc,+id+asc&group.limit=10&distrib=false&group.distributed.second=true&wt=javabin&rows=100&f.a_t.facet.limit=160&version=2&fl=id&shard.url=[ff01::083]:33332/solr|127.0.0.1:54308/solr&NOW=1352423576528&start=0&q=*:*&group.field=a_si&group=true&facet.field=a_t&isShard=true} status=0 QTime=16 
[junit4:junit4]   2> 4907 T170 C3 REQ [collection1] webapp=/solr path=/select params={facet=true&sort=a_si+asc,+id+asc&group.limit=10&wt=javabin&rows=100&version=2&fl=id,a_si&shards=[ff01::083]:33332/solr|127.0.0.1:54308/solr&q=*:*&group.field=a_si&group=true&facet.field=a_t} status=0 QTime=58 
[junit4:junit4]   2> 4953 T156 C2 REQ [collection1] webapp=/solr path=/select params={sort=a_si+asc,+id+asc&stats=true&group.limit=10&distrib=false&wt=javabin&rows=100&version=2&fl=id,a_si&q=*:*&stats.field=a_si&group.field=a_si&group=true} hits=0 status=0 QTime=43 
[junit4:junit4]   2> 4962 T170 C3 REQ [collection1] webapp=/solr path=/select params={sort=a_si+asc,+id+asc&group.distributed.first=true&stats=false&group.limit=10&distrib=false&wt=javabin&rows=100&version=2&fl=id&shard.url=127.0.0.1:54308/solr|[ff01::213]:33332/solr|[ff01::114]:33332/solr&NOW=1352423576635&start=0&q=*:*&stats.field=a_si&group.field=a_si&group=true&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 4971 T170 C3 REQ [collection1] webapp=/solr path=/select params={sort=a_si+asc,+id+asc&stats=true&group.limit=10&distrib=false&group.distributed.second=true&wt=javabin&rows=100&version=2&fl=id&shard.url=127.0.0.1:54308/solr|[ff01::213]:33332/solr|[ff01::114]:33332/solr&NOW=1352423576635&start=0&q=*:*&stats.field=a_si&group.field=a_si&group=true&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 4972 T167 C3 REQ [collection1] webapp=/solr path=/select params={sort=a_si+asc,+id+asc&stats=true&group.limit=10&wt=javabin&rows=100&version=2&fl=id,a_si&shards=127.0.0.1:54308/solr|[ff01::213]:33332/solr|[ff01::114]:33332/solr&q=*:*&stats.field=a_si&group.field=a_si&group=true} status=0 QTime=16 
[junit4:junit4]   2> 5076 T155 C2 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&sort=a_si+asc,+id+asc&group.limit=10&distrib=false&wt=javabin&version=2&rows=100&fl=id,a_si&q=kings&group.field=a_si&group=true&qt=spellCheckCompRH&spellcheck.build=true} hits=0 status=0 QTime=100 
[junit4:junit4]   2> 5100 T167 C3 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&sort=a_si+asc,+id+asc&group.distributed.first=true&spellcheck.extendedResults=false&group.limit=10&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&rows=100&version=2&fl=id&shard.url=127.0.0.1:54308/solr|[ff01::114]:33332/solr&NOW=1352423576762&start=0&q=kings&group.field=a_si&spellcheck.count=1&group=true&isShard=true&spellcheck.build=true} status=0 QTime=1 
[junit4:junit4]   2> 5105 T170 C3 oashc.SpellCheckComponent.finishStage 127.0.0.1:54308/solr|[ff01::114]:33332/solr null
[junit4:junit4]   2> 5105 T170 C3 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&sort=a_si+asc,+id+asc&group.limit=10&wt=javabin&version=2&rows=100&shards=127.0.0.1:54308/solr|[ff01::114]:33332/solr&fl=id,a_si&q=kings&group.field=a_si&group=true&qt=spellCheckCompRH&spellcheck.build=true} status=0 QTime=22 
[junit4:junit4]   2> 5167 T153 C2 REQ [collection1] webapp=/solr path=/select params={facet=true&sort=a_si+asc,+id+asc&group.limit=10&distrib=false&wt=javabin&rows=100&version=2&fl=id,a_si&q=*:*&group.truncate=true&group.field=a_si&group=true&facet.field=a_t&fq=a_s:a} hits=0 status=0 QTime=59 
[junit4:junit4]   2> 5174 T170 C3 REQ [collection1] webapp=/solr path=/select params={facet=false&sort=a_si+asc,+id+asc&group.distributed.first=true&group.limit=10&distrib=false&wt=javabin&rows=100&version=2&fl=id&shard.url=[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:54308/solr|[ff01::213]:33332/solr&NOW=1352423576849&start=0&q=*:*&group.truncate=true&group.field=a_si&group=true&facet.field=a_t&isShard=true&fq=a_s:a} status=0 QTime=0 
[junit4:junit4]   2> 5215 T170 C3 REQ [collection1] webapp=/solr path=/select params={facet=true&sort=a_si+asc,+id+asc&group.limit=10&distrib=false&group.distributed.second=true&wt=javabin&rows=100&f.a_t.facet.limit=160&version=2&fl=id&shard.url=[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:54308/solr|[ff01::213]:33332/solr&NOW=1352423576849&start=0&q=*:*&group.truncate=true&group.field=a_si&group=true&facet.field=a_t&isShard=true&fq=a_s:a} status=0 QTime=36 
[junit4:junit4]   2> 5216 T167 C3 REQ [collection1] webapp=/solr path=/select params={facet=true&sort=a_si+asc,+id+asc&group.limit=10&wt=javabin&rows=100&version=2&fl=id,a_si&shards=[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:54308/solr|[ff01::213]:33332/solr&q=*:*&group.truncate=true&group.field=a_si&group=true&facet.field=a_t&fq=a_s:a} status=0 QTime=46 
[junit4:junit4]   2> 5231 T155 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 10
[junit4:junit4]   2> 5239 T170 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 4
[junit4:junit4]   2> 5250 T153 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 6
[junit4:junit4]   2> 5259 T167 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 6
[junit4:junit4]   2> 5268 T156 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 5
[junit4:junit4]   2> 5276 T170 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 4
[junit4:junit4]   2> 5284 T155 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 3
[junit4:junit4]   2> 5291 T167 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 4
[junit4:junit4]   2> 5297 T153 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 3
[junit4:junit4]   2> 5303 T170 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 3
[junit4:junit4]   2> 5309 T156 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 3
[junit4:junit4]   2> 5315 T167 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 3
[junit4:junit4]   2> 5321 T155 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 3
[junit4:junit4]   2> 5327 T170 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 3
[junit4:junit4]   2> 5333 T153 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 3
[junit4:junit4]   2> 5339 T167 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 3
[junit4:junit4]   2> 5345 T156 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 3
[junit4:junit4]   2> 5351 T170 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 3
[junit4:junit4]   2> 5392 T155 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 38
[junit4:junit4]   2> 5423 T167 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 26
[junit4:junit4]   2> 5430 T153 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[11]} 0 4
[junit4:junit4]   2> 5437 T170 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[11]} 0 4
[junit4:junit4]   2> 5443 T156 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 3
[junit4:junit4]   2> 5450 T167 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 3
[junit4:junit4]   2> 5456 T155 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 2
[junit4:junit4]   2> 5462 T170 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 2
[junit4:junit4]   2> 5468 T153 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 2
[junit4:junit4]   2> 5474 T167 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 3
[junit4:junit4]   2> 5480 T156 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 3
[junit4:junit4]   2> 5487 T170 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 2
[junit4:junit4]   2> 5506 T155 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 14
[junit4:junit4]   2> 5519 T167 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 8
[junit4:junit4]   2> 5527 T153 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 3
[junit4:junit4]   2> 5536

[...truncated too long message...]

[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=184, name=qtp2042731979-184 Acceptor1 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=203, name=qtp1894659009-203 Acceptor2 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    9) Thread[id=202, name=qtp1894659009-202 Acceptor1 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   10) Thread[id=236, name=qtp1558648238-236 Acceptor3 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=183, name=qtp2042731979-183 Acceptor0 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=185, name=qtp2042731979-185 Acceptor2 SelectChannelConnector@0.0.0.0:22639, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=233, name=qtp1558648238-233 Acceptor0 SelectChannelConnector@0.0.0.0:21320, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=204, name=qtp1894659009-204 Acceptor3 SelectChannelConnector@0.0.0.0:47890, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=234, name=qtp1558648238-234 Acceptor1 SelectChannelConnector@0.0.0.0:21320, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=254, name=qtp604251701-254 Acceptor3 SelectChannelConnector@0.0.0.0:12486, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([8C944CC3AC69D41E]:0)
[junit4:junit4] Completed on J0 in 328.42s, 1 test, 2 errors <<< FAILURES!

[...truncated 107 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 970 tests, 4 suite-level errors, 798 ignored (2 assumptions)

Total time: 33 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 838 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/838/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test:     1) Thread[id=637, name=qtp1281562065-637 Acceptor0 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=489, name=qtp36077817-489 Acceptor1 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=638, name=qtp1281562065-638 Acceptor1 SelectChannelConnector@0.0.0.0:18988, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=507, name=qtp66767496-507 Acceptor1 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=490, name=qtp36077817-490 Acceptor2 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=639, name=qtp1281562065-639 Acceptor2 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=438, name=qtp1811684413-438 Acceptor0 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=640, name=qtp1281562065-640 Acceptor3 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=509, name=qtp66767496-509 Acceptor3 SelectChannelConnector@0.0.0.0:65134, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=439, name=qtp1811684413-439 Acceptor1 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=491, name=qtp36077817-491 Acceptor3 SelectChannelConnector@0.0.0.0:32669, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=506, name=qtp66767496-506 Acceptor0 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=440, name=qtp1811684413-440 Acceptor2 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=488, name=qtp36077817-488 Acceptor0 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=508, name=qtp66767496-508 Acceptor2 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=441, name=qtp1811684413-441 Acceptor3 SelectChannelConnector@0.0.0.0:16546, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test: 
   1) Thread[id=637, name=qtp1281562065-637 Acceptor0 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=489, name=qtp36077817-489 Acceptor1 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=638, name=qtp1281562065-638 Acceptor1 SelectChannelConnector@0.0.0.0:18988, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=507, name=qtp66767496-507 Acceptor1 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=490, name=qtp36077817-490 Acceptor2 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=639, name=qtp1281562065-639 Acceptor2 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=438, name=qtp1811684413-438 Acceptor0 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=640, name=qtp1281562065-640 Acceptor3 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=509, name=qtp66767496-509 Acceptor3 SelectChannelConnector@0.0.0.0:65134, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=439, name=qtp1811684413-439 Acceptor1 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=491, name=qtp36077817-491 Acceptor3 SelectChannelConnector@0.0.0.0:32669, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=506, name=qtp66767496-506 Acceptor0 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=440, name=qtp1811684413-440 Acceptor2 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=488, name=qtp36077817-488 Acceptor0 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=508, name=qtp66767496-508 Acceptor2 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=441, name=qtp1811684413-441 Acceptor3 SelectChannelConnector@0.0.0.0:16546, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([FF514C3648EA9C60]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=637, name=qtp1281562065-637 Acceptor0 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=489, name=qtp36077817-489 Acceptor1 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=638, name=qtp1281562065-638 Acceptor1 SelectChannelConnector@0.0.0.0:18988, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=507, name=qtp66767496-507 Acceptor1 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=490, name=qtp36077817-490 Acceptor2 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=639, name=qtp1281562065-639 Acceptor2 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=438, name=qtp1811684413-438 Acceptor0 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=509, name=qtp66767496-509 Acceptor3 SelectChannelConnector@0.0.0.0:65134, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=640, name=qtp1281562065-640 Acceptor3 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=439, name=qtp1811684413-439 Acceptor1 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=491, name=qtp36077817-491 Acceptor3 SelectChannelConnector@0.0.0.0:32669, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=506, name=qtp66767496-506 Acceptor0 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=440, name=qtp1811684413-440 Acceptor2 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=488, name=qtp36077817-488 Acceptor0 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=508, name=qtp66767496-508 Acceptor2 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=441, name=qtp1811684413-441 Acceptor3 SelectChannelConnector@0.0.0.0:16546, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=637, name=qtp1281562065-637 Acceptor0 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=489, name=qtp36077817-489 Acceptor1 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=638, name=qtp1281562065-638 Acceptor1 SelectChannelConnector@0.0.0.0:18988, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=507, name=qtp66767496-507 Acceptor1 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=490, name=qtp36077817-490 Acceptor2 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=639, name=qtp1281562065-639 Acceptor2 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=438, name=qtp1811684413-438 Acceptor0 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=509, name=qtp66767496-509 Acceptor3 SelectChannelConnector@0.0.0.0:65134, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=640, name=qtp1281562065-640 Acceptor3 SelectChannelConnector@0.0.0.0:18988, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=439, name=qtp1811684413-439 Acceptor1 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=491, name=qtp36077817-491 Acceptor3 SelectChannelConnector@0.0.0.0:32669, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=506, name=qtp66767496-506 Acceptor0 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=440, name=qtp1811684413-440 Acceptor2 SelectChannelConnector@0.0.0.0:16546, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=488, name=qtp36077817-488 Acceptor0 SelectChannelConnector@0.0.0.0:32669, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=508, name=qtp66767496-508 Acceptor2 SelectChannelConnector@0.0.0.0:65134, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=441, name=qtp1811684413-441 Acceptor3 SelectChannelConnector@0.0.0.0:16546, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([FF514C3648EA9C60]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([FF514C3648EA9C60]:0)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([FF514C3648EA9C60]:0)




Build Log:
[...truncated 7861 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4]   2> 4 T422 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1352300491962
[junit4:junit4]   2> 7 T422 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 15 T423 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 16 T423 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 17 T423 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 113 T422 oasc.ZkTestServer.run start zk server on port:17418
[junit4:junit4]   2> 114 T422 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:17418 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@395e4874
[junit4:junit4]   2> 115 T428 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:17418
[junit4:junit4]   2> 115 T422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 116 T428 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:17418, initiating session
[junit4:junit4]   2> 116 T424 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:59503
[junit4:junit4]   2> 117 T424 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:59503
[junit4:junit4]   2> 117 T426 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 120 T426 oazs.NIOServerCnxn.finishSessionInit Established session 0x13adb63f4c60000 with negotiated timeout 10000 for client /140.211.11.196:59503
[junit4:junit4]   2> 120 T428 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:17418, sessionid = 0x13adb63f4c60000, negotiated timeout = 10000
[junit4:junit4]   2> 120 T429 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@395e4874 name:ZooKeeperConnection Watcher:127.0.0.1:17418 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 120 T422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 120 T422 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 124 T427 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13adb63f4c60000
[junit4:junit4]   2> 148 T429 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 149 T424 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:59503 which had sessionid 0x13adb63f4c60000
[junit4:junit4]   2> 148 T422 oaz.ZooKeeper.close Session: 0x13adb63f4c60000 closed
[junit4:junit4]   2> 149 T422 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:17418/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@4807767e
[junit4:junit4]   2> 150 T430 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:17418
[junit4:junit4]   2> 151 T422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 151 T430 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:17418, initiating session
[junit4:junit4]   2> 152 T424 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:55975
[junit4:junit4]   2> 152 T424 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:55975
[junit4:junit4]   2> 153 T426 oazs.NIOServerCnxn.finishSessionInit Established session 0x13adb63f4c60001 with negotiated timeout 10000 for client /140.211.11.196:55975
[junit4:junit4]   2> 153 T430 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:17418, sessionid = 0x13adb63f4c60001, negotiated timeout = 10000
[junit4:junit4]   2> 154 T431 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4807767e name:ZooKeeperConnection Watcher:127.0.0.1:17418/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 154 T422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 154 T422 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 170 T422 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 175 T422 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 177 T422 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 181 T422 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 182 T422 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 188 T422 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 189 T422 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 293 T422 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 294 T422 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 317 T422 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 318 T422 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 321 T422 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 322 T422 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 376 T422 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 377 T422 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 381 T422 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 381 T422 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 385 T422 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 385 T422 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 417 T422 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 417 T422 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 421 T422 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 422 T422 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 426 T427 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13adb63f4c60001
[junit4:junit4]   2> 431 T431 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 431 T424 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:55975 which had sessionid 0x13adb63f4c60001
[junit4:junit4]   2> 431 T422 oaz.ZooKeeper.close Session: 0x13adb63f4c60001 closed
[junit4:junit4]   2> 1010 T422 oejs.Server.doStart jetty-8.1.7.v20120910
[junit4:junit4]   2> 1366 T422 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:16546
[junit4:junit4]   2> 1412 T422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1413 T422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1413 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1457 T422 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1457 T422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1458 T422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1458 T422 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1459 T422 oasc.CoreContainer.<init> New CoreContainer 1320953870
[junit4:junit4]   2> 1459 T422 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1460 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1486 T422 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1501 T422 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:17418/solr
[junit4:junit4]   2> 1501 T422 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:17418/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@45e7753c
[junit4:junit4]   2> 1502 T442 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:17418
[junit4:junit4]   2> 1502 T422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1504 T442 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:17418, initiating session
[junit4:junit4]   2> 1504 T424 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:20178
[junit4:junit4]   2> 1504 T424 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:20178
[junit4:junit4]   2> 1506 T426 oazs.NIOServerCnxn.finishSessionInit Established session 0x13adb63f4c60002 with negotiated timeout 5000 for client /140.211.11.196:20178
[junit4:junit4]   2> 1506 T442 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:17418, sessionid = 0x13adb63f4c60002, negotiated timeout = 5000
[junit4:junit4]   2> 1507 T443 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45e7753c name:ZooKeeperConnection Watcher:127.0.0.1:17418/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1507 T422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1509 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1528 T422 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1530 T422 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:16546_solr
[junit4:junit4]   2> 1531 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:delete cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:16546_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:16546_solr
[junit4:junit4]   2> 1537 T422 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:16546_solr
[junit4:junit4]   2> 1540 T422 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1540 T422 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1541 T422 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1541 T422 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1542 T422 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1542 T422 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1543 T422 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1543 T422 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1544 T422 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1544 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1562 T422 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1580 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:delete cxid:0x15 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1582 T422 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1591 T422 oasc.Overseer.start Overseer (id=88624365042008066-127.0.0.1:16546_solr-n_0000000000) starting
[junit4:junit4]   2> 1592 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1593 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1595 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:create cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1612 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1613 T445 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1613 T422 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1626 T422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1637 T444 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1641 T422 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1642 T422 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 1643 T422 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1643 T422 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 1645 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1647 T422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1647 T422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1705 T422 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1779 T422 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1880 T422 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1889 T422 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2494 T422 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2507 T422 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2511 T422 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2531 T422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2536 T422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2541 T422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2543 T422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2545 T422 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/control/data/
[junit4:junit4]   2> 2545 T422 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@53ea0105
[junit4:junit4]   2> 2597 T422 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/control/data/index/
[junit4:junit4]   2> 2598 T422 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2599 T422 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/control/data/index forceNew:false
[junit4:junit4]   2> 2607 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/control/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2607 T422 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2608 T422 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 2610 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2611 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 2611 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2612 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 2613 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2613 T422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2614 T422 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2614 T422 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2615 T422 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2615 T422 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 2616 T422 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 2617 T422 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2617 T422 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2618 T422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2618 T422 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2619 T422 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2619 T422 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2620 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2620 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2621 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2622 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2622 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2623 T422 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2623 T422 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 2624 T422 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 2633 T422 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2637 T422 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 2638 T422 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 2643 T422 oass.SolrIndexSearcher.<init> Opening Searcher@7771652f main
[junit4:junit4]   2> 2655 T422 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2655 T422 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2656 T422 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 2680 T422 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2750 T450 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7771652f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2754 T422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2755 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:create cxid:0x56 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2772 T422 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2773 T422 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:16546/solr shardId:control_shard
[junit4:junit4]   2> 2774 T422 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/control_shard/election
[junit4:junit4]   2> 2790 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:delete cxid:0x65 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2790 T422 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2791 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:create cxid:0x66 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3146 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3147 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:16546_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:16546/solr"}
[junit4:junit4]   2> 3148 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:create cxid:0x6d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 3171 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3300 T422 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 3301 T422 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 3301 T422 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:16546/solr/collection1/
[junit4:junit4]   2> 3302 T422 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 3303 T422 oasc.SyncStrategy.syncToMe http://127.0.0.1:16546/solr/collection1/ has no replicas
[junit4:junit4]   2> 3303 T422 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:16546/solr/collection1/
[junit4:junit4]   2> 3303 T422 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/control_shard
[junit4:junit4]   2> 3309 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60002 type:create cxid:0x8b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3686 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3700 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3730 T422 oasc.ZkController.register We are http://127.0.0.1:16546/solr/collection1/ and leader is http://127.0.0.1:16546/solr/collection1/
[junit4:junit4]   2> 3737 T422 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:16546/solr
[junit4:junit4]   2> 3737 T422 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3737 T422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3741 T422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3742 T422 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3743 T422 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3832 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4058 T422 oejs.Server.doStart jetty-8.1.7.v20120910
[junit4:junit4]   2> 4061 T422 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:65407
[junit4:junit4]   2> 4062 T422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4063 T422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty1-1352300495823
[junit4:junit4]   2> 4063 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty1-1352300495823/'
[junit4:junit4]   2> 4105 T422 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4105 T422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4106 T422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty1-1352300495823
[junit4:junit4]   2> 4106 T422 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty1-1352300495823/solr.xml
[junit4:junit4]   2> 4106 T422 oasc.CoreContainer.<init> New CoreContainer 1803410626
[junit4:junit4]   2> 4107 T422 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty1-1352300495823/'
[junit4:junit4]   2> 4108 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty1-1352300495823/'
[junit4:junit4]   2> 4134 T422 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4150 T422 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:17418/solr
[junit4:junit4]   2> 4151 T422 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:17418/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@6dd06a26
[junit4:junit4]   2> 4152 T460 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:17418
[junit4:junit4]   2> 4152 T422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4153 T460 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:17418, initiating session
[junit4:junit4]   2> 4153 T424 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:36132
[junit4:junit4]   2> 4154 T424 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:36132
[junit4:junit4]   2> 4157 T426 oazs.NIOServerCnxn.finishSessionInit Established session 0x13adb63f4c60003 with negotiated timeout 5000 for client /140.211.11.196:36132
[junit4:junit4]   2> 4157 T460 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:17418, sessionid = 0x13adb63f4c60003, negotiated timeout = 5000
[junit4:junit4]   2> 4158 T461 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6dd06a26 name:ZooKeeperConnection Watcher:127.0.0.1:17418/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4158 T422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4159 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60003 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4172 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4175 T422 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65407_solr
[junit4:junit4]   2> 4176 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60003 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:65407_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:65407_solr
[junit4:junit4]   2> 4178 T422 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65407_solr
[junit4:junit4]   2> 4188 T443 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4188 T422 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 4189 T422 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4189 T422 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 4190 T422 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4190 T422 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4191 T422 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4191 T422 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4191 T422 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4192 T422 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4192 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 4206 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4206 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:16546_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:16546/solr"}
[junit4:junit4]   2> 4218 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 4219 T422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 4235 T422 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty1-1352300495823/collection1
[junit4:junit4]   2> 4235 T422 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 4236 T422 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 4236 T422 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 4238 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty1-1352300495823/collection1/'
[junit4:junit4]   2> 4240 T422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-jetty1-1352300495823/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 4241 T422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-jetty1-1352300495823/collection1/lib/README' to classloader
[junit4:junit4]   2> 4301 T422 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 4374 T422 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 4475 T422 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 4484 T422 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 5134 T422 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5148 T422 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5152 T422 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5172 T422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5177 T422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5183 T422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5184 T422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5186 T422 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty1-1352300495823/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty1/
[junit4:junit4]   2> 5186 T422 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@53ea0105
[junit4:junit4]   2> 5188 T422 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty1/index/
[junit4:junit4]   2> 5189 T422 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5190 T422 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty1/index forceNew:false
[junit4:junit4]   2> 5228 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty1/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5229 T422 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5230 T422 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 5232 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5233 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 5233 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 5234 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 5235 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 5235 T422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 5236 T422 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 5237 T422 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5237 T422 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5238 T422 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 5239 T422 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 5239 T422 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5240 T422 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 5240 T422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 5241 T422 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 5242 T422 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5242 T422 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5243 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5244 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5244 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5245 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5245 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5246 T422 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5247 T422 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 5247 T422 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 5257 T422 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5263 T422 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 5264 T422 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 5269 T422 oass.SolrIndexSearcher.<init> Opening Searcher@38936974 main
[junit4:junit4]   2> 5270 T422 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5271 T422 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5271 T422 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 5329 T422 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 5443 T466 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@38936974 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5448 T422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 5735 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5736 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:65407_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:65407/solr"}
[junit4:junit4]   2> 5736 T444 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 5750 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5750 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5951 T422 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 5951 T422 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:65407/solr shardId:shard1
[junit4:junit4]   2> 5952 T422 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 5971 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60003 type:delete cxid:0x42 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 5978 T422 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 5978 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60003 type:create cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5981 T422 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 5981 T422 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 5981 T422 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:65407/solr/collection1/
[junit4:junit4]   2> 5982 T422 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 5982 T422 oasc.SyncStrategy.syncToMe http://127.0.0.1:65407/solr/collection1/ has no replicas
[junit4:junit4]   2> 5982 T422 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:65407/solr/collection1/
[junit4:junit4]   2> 5983 T422 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 5992 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60003 type:create cxid:0x4d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6256 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6285 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6285 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6307 T422 oasc.ZkController.register We are http://127.0.0.1:65407/solr/collection1/ and leader is http://127.0.0.1:65407/solr/collection1/
[junit4:junit4]   2> 6307 T422 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:65407/solr
[junit4:junit4]   2> 6307 T422 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 6308 T422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6309 T422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6310 T422 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 6310 T422 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 6311 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 6450 T422 oejs.Server.doStart jetty-8.1.7.v20120910
[junit4:junit4]   2> 6452 T422 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:23255
[junit4:junit4]   2> 6453 T422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6454 T422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty2-1352300498279
[junit4:junit4]   2> 6454 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty2-1352300498279/'
[junit4:junit4]   2> 6485 T422 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 6486 T422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6486 T422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty2-1352300498279
[junit4:junit4]   2> 6486 T422 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty2-1352300498279/solr.xml
[junit4:junit4]   2> 6486 T422 oasc.CoreContainer.<init> New CoreContainer 782077563
[junit4:junit4]   2> 6487 T422 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty2-1352300498279/'
[junit4:junit4]   2> 6487 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty2-1352300498279/'
[junit4:junit4]   2> 6507 T422 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 6519 T422 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:17418/solr
[junit4:junit4]   2> 6519 T422 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:17418/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@658eb67b
[junit4:junit4]   2> 6520 T476 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:17418
[junit4:junit4]   2> 6520 T422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6521 T476 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:17418, initiating session
[junit4:junit4]   2> 6522 T424 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:59780
[junit4:junit4]   2> 6522 T424 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:59780
[junit4:junit4]   2> 6525 T426 oazs.NIOServerCnxn.finishSessionInit Established session 0x13adb63f4c60004 with negotiated timeout 5000 for client /140.211.11.196:59780
[junit4:junit4]   2> 6525 T476 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:17418, sessionid = 0x13adb63f4c60004, negotiated timeout = 5000
[junit4:junit4]   2> 6525 T477 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@658eb67b name:ZooKeeperConnection Watcher:127.0.0.1:17418/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6525 T422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6526 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60004 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6537 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60004 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6540 T422 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:23255_solr
[junit4:junit4]   2> 6540 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60004 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:23255_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:23255_solr
[junit4:junit4]   2> 6542 T422 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:23255_solr
[junit4:junit4]   2> 6552 T461 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 6553 T443 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 6553 T422 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 6554 T422 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 6554 T422 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 6554 T422 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 6554 T422 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 6555 T422 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 6555 T422 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 6555 T422 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 6556 T422 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 6556 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 6573 T422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 6578 T422 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty2-1352300498279/collection1
[junit4:junit4]   2> 6579 T422 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 6579 T422 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 6580 T422 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 6581 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty2-1352300498279/collection1/'
[junit4:junit4]   2> 6582 T422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-jetty2-1352300498279/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 6583 T422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-jetty2-1352300498279/collection1/lib/README' to classloader
[junit4:junit4]   2> 6624 T422 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 6672 T422 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 6773 T422 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 6779 T422 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6803 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6804 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:65407_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:65407/solr"}
[junit4:junit4]   2> 6815 T477 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6815 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6815 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7232 T422 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 7241 T422 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 7244 T422 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 7259 T422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7263 T422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7267 T422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7268 T422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7269 T422 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty2-1352300498279/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty2/
[junit4:junit4]   2> 7270 T422 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@53ea0105
[junit4:junit4]   2> 7271 T422 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty2/index/
[junit4:junit4]   2> 7271 T422 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 7272 T422 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty2/index forceNew:false
[junit4:junit4]   2> 7277 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty2/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 7278 T422 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 7278 T422 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 7280 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 7280 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 7281 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 7281 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 7282 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 7282 T422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 7282 T422 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 7283 T422 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 7283 T422 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 7284 T422 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 7284 T422 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 7285 T422 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 7285 T422 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 7285 T422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 7286 T422 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 7286 T422 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 7286 T422 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7287 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7287 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7288 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7288 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7289 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7289 T422 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7290 T422 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 7290 T422 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 7301 T422 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 7305 T422 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 7306 T422 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 7309 T422 oass.SolrIndexSearcher.<init> Opening Searcher@2b06a66b main
[junit4:junit4]   2> 7310 T422 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 7310 T422 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7311 T422 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 7317 T422 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 7386 T482 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2b06a66b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 7388 T422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7823 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7823 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:23255_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:23255/solr"}
[junit4:junit4]   2> 7837 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7837 T477 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7837 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7890 T422 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 7890 T422 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:23255/solr shardId:shard2
[junit4:junit4]   2> 7891 T422 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 7909 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60004 type:delete cxid:0x42 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 7910 T422 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7910 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60004 type:create cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7913 T422 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7913 T422 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7913 T422 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:23255/solr/collection1/
[junit4:junit4]   2> 7914 T422 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7914 T422 oasc.SyncStrategy.syncToMe http://127.0.0.1:23255/solr/collection1/ has no replicas
[junit4:junit4]   2> 7915 T422 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:23255/solr/collection1/
[junit4:junit4]   2> 7915 T422 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 7919 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60004 type:create cxid:0x4c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8342 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8370 T477 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8370 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8370 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8380 T422 oasc.ZkController.register We are http://127.0.0.1:23255/solr/collection1/ and leader is http://127.0.0.1:23255/solr/collection1/
[junit4:junit4]   2> 8380 T422 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:23255/solr
[junit4:junit4]   2> 8380 T422 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8381 T422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8382 T422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8383 T422 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 8383 T422 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8384 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8530 T422 oejs.Server.doStart jetty-8.1.7.v20120910
[junit4:junit4]   2> 8533 T422 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:32669
[junit4:junit4]   2> 8534 T422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8534 T422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty3-1352300500350
[junit4:junit4]   2> 8534 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty3-1352300500350/'
[junit4:junit4]   2> 8565 T422 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8565 T422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8566 T422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty3-1352300500350
[junit4:junit4]   2> 8566 T422 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty3-1352300500350/solr.xml
[junit4:junit4]   2> 8566 T422 oasc.CoreContainer.<init> New CoreContainer 594667272
[junit4:junit4]   2> 8567 T422 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty3-1352300500350/'
[junit4:junit4]   2> 8567 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty3-1352300500350/'
[junit4:junit4]   2> 8592 T422 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8603 T422 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:17418/solr
[junit4:junit4]   2> 8604 T422 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:17418/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@589e3112
[junit4:junit4]   2> 8604 T492 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:17418
[junit4:junit4]   2> 8604 T422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8606 T492 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:17418, initiating session
[junit4:junit4]   2> 8606 T424 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:53853
[junit4:junit4]   2> 8606 T424 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:53853
[junit4:junit4]   2> 8609 T426 oazs.NIOServerCnxn.finishSessionInit Established session 0x13adb63f4c60005 with negotiated timeout 5000 for client /140.211.11.196:53853
[junit4:junit4]   2> 8609 T492 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:17418, sessionid = 0x13adb63f4c60005, negotiated timeout = 5000
[junit4:junit4]   2> 8609 T493 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@589e3112 name:ZooKeeperConnection Watcher:127.0.0.1:17418/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8609 T422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8610 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60005 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8623 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60005 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8624 T422 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32669_solr
[junit4:junit4]   2> 8625 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:32669_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:32669_solr
[junit4:junit4]   2> 8631 T422 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32669_solr
[junit4:junit4]   2> 8634 T443 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 8634 T477 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 8634 T461 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 8634 T422 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 8635 T422 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8635 T422 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 8636 T422 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8636 T422 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8637 T422 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8637 T422 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8637 T422 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8638 T422 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8638 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 8659 T422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 8665 T422 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty3-1352300500350/collection1
[junit4:junit4]   2> 8666 T422 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 8667 T422 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 8667 T422 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 8669 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty3-1352300500350/collection1/'
[junit4:junit4]   2> 8670 T422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-jetty3-1352300500350/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 8671 T422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-jetty3-1352300500350/collection1/lib/README' to classloader
[junit4:junit4]   2> 8723 T422 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 8793 T422 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 8877 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8878 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:23255_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:23255/solr"}
[junit4:junit4]   2> 8894 T422 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 8900 T477 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 8901 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 8901 T493 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 8901 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 8907 T422 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 9526 T422 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 9540 T422 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 9545 T422 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 9565 T422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9571 T422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9576 T422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9577 T422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9579 T422 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty3-1352300500350/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/
[junit4:junit4]   2> 9579 T422 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@53ea0105
[junit4:junit4]   2> 9581 T422 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index/
[junit4:junit4]   2> 9582 T422 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 9583 T422 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index forceNew:false
[junit4:junit4]   2> 9589 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9590 T422 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 9591 T422 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 9593 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 9594 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 9594 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 9595 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 9596 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 9596 T422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 9597 T422 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 9597 T422 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 9598 T422 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 9599 T422 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 9599 T422 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 9600 T422 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 9601 T422 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 9601 T422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 9602 T422 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 9602 T422 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 9603 T422 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9604 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9604 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9605 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9606 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9606 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9607 T422 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9608 T422 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 9608 T422 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 9617 T422 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 9623 T422 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 9625 T422 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 9629 T422 oass.SolrIndexSearcher.<init> Opening Searcher@73602ff8 main
[junit4:junit4]   2> 9631 T422 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 9631 T422 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 9632 T422 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 9641 T422 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 9697 T498 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73602ff8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 9702 T422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 9909 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9910 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:32669_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:32669/solr"}
[junit4:junit4]   2> 9934 T477 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9934 T493 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9934 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9934 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10205 T422 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 10205 T422 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:32669/solr shardId:shard1
[junit4:junit4]   2> 10209 T422 oasc.ZkController.register We are http://127.0.0.1:32669/solr/collection1/ and leader is http://127.0.0.1:65407/solr/collection1/
[junit4:junit4]   2> 10209 T422 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32669/solr
[junit4:junit4]   2> 10210 T422 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 10210 T422 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C10 name=collection1 org.apache.solr.core.SolrCore@21a95d71 url=http://127.0.0.1:32669/solr/collection1 node=127.0.0.1:32669_solr
[junit4:junit4]   2> 10218 T499 C10 P32669 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 10219 T422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10219 T499 C10 P32669 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 10219 T499 C10 P32669 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 10220 T422 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 10221 T422 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 10221 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 10234 T499 C10 P32669 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:65407/solr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 10243 T499 C10 P32669 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 10258 T499 C10 P32669 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:32669/solr START replicas=[http://127.0.0.1:65407/solr/collection1/] nUpdates=100
[junit4:junit4]   2> 10270 T499 C10 P32669 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 10271 T499 C10 P32669 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 10271 T499 C10 P32669 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 10271 T499 C10 P32669 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 10440 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10441 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:32669_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:32669/solr"}
[junit4:junit4]   2> 10460 T477 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10460 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10461 T493 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10461 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10465 T422 oejs.Server.doStart jetty-8.1.7.v20120910
[junit4:junit4]   2> 10468 T422 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:65134
[junit4:junit4]   2> 10469 T422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 10470 T422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty4-1352300502195
[junit4:junit4]   2> 10470 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty4-1352300502195/'
[junit4:junit4]   2> 10690 T422 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 10690 T422 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 10691 T422 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty4-1352300502195
[junit4:junit4]   2> 10691 T422 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty4-1352300502195/solr.xml
[junit4:junit4]   2> 10692 T422 oasc.CoreContainer.<init> New CoreContainer 1325726217
[junit4:junit4]   2> 10693 T422 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty4-1352300502195/'
[junit4:junit4]   2> 10693 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty4-1352300502195/'
[junit4:junit4]   2> 11002 T422 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 11127 T422 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:17418/solr
[junit4:junit4]   2> 11127 T422 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:17418/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@3c05a2f2
[junit4:junit4]   2> 11129 T514 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:17418
[junit4:junit4]   2> 11129 T422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 11130 T514 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:17418, initiating session
[junit4:junit4]   2> 11130 T424 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:11296
[junit4:junit4]   2> 11131 T424 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:11296
[junit4:junit4]   2> 11132 T426 oazs.NIOServerCnxn.finishSessionInit Established session 0x13adb63f4c60006 with negotiated timeout 5000 for client /140.211.11.196:11296
[junit4:junit4]   2> 11132 T514 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:17418, sessionid = 0x13adb63f4c60006, negotiated timeout = 5000
[junit4:junit4]   2> 11133 T515 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c05a2f2 name:ZooKeeperConnection Watcher:127.0.0.1:17418/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 11133 T422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 11134 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60006 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11135 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60006 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11144 T422 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65134_solr
[junit4:junit4]   2> 11144 T427 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13adb63f4c60006 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:65134_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:65134_solr
[junit4:junit4]   2> 11145 T422 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:65134_solr
[junit4:junit4]   2> 11149 T477 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11149 T493 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11149 T443 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11149 T461 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11150 T422 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 11150 T422 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 11151 T422 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 11151 T422 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 11151 T422 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 11152 T422 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 11152 T422 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 11153 T422 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 11153 T422 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 11153 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 11177 T422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 11217 T422 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty4-1352300502195/collection1
[junit4:junit4]   2> 11218 T422 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 11220 T422 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 11220 T422 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 11222 T422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty4-1352300502195/collection1/'
[junit4:junit4]   2> 11223 T422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-jetty4-1352300502195/collection1/lib/README' to classloader
[junit4:junit4]   2> 11224 T422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-jetty4-1352300502195/collection1/lib/classes/' to classloader
[junit4:junit4]   2> ASYNC  NEW_CORE C11 name=collection1 org.apache.solr.core.SolrCore@75885ad3 url=http://127.0.0.1:65407/solr/collection1 node=127.0.0.1:65407_solr
[junit4:junit4]   2> 11627 T463 C11 P65407 REQ /get {distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=14 
[junit4:junit4]   2> 11649 T422 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 11805 T422 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 11906 T422 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 11914 T422 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 12338 T422 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 12347 T422 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 12350 T422 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 12367 T422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 12372 T422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 12376 T422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 12377 T422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 12379 T422 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-jetty4-1352300502195/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty4/
[junit4:junit4]   2> 12379 T422 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@53ea0105
[junit4:junit4]   2> 12380 T422 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty4/index/
[junit4:junit4]   2> 12381 T422 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 12381 T422 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty4/index forceNew:false
[junit4:junit4]   2> 12386 T422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty4/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 12386 T422 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 12386 T422 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 12388 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 12388 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 12389 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 12389 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 12390 T422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 12390 T422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 12391 T422 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 12391 T422 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 12391 T422 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 12392 T422 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 12392 T422 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 12393 T422 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 12393 T422 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 12394 T422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 12394 T422 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 12394 T422 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 12395 T422 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 12395 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 12396 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 12396 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 12397 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 12397 T422 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 12398 T422 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 12398 T422 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 12399 T422 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 12404 T422 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 12407 T422 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 12408 T422 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 12411 T422 oass.SolrIndexSearcher.<init> Opening Searcher@7aea8a58 main
[junit4:junit4]   2> 12412 T422 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 12413 T422 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 12413 T422 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 12419 T422 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 12467 T516 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7aea8a58 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 12469 T422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 12473 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 12474 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:65134_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:65134/solr"}
[junit4:junit4]   2> 12484 T477 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 12484 T515 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 12484 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 12484 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 12484 T493 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 12972 T422 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 12972 T422 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:65134/solr shardId:shard2
[junit4:junit4]   2> 12976 T422 oasc.ZkController.register We are http://127.0.0.1:65134/solr/collection1/ and leader is http://127.0.0.1:23255/solr/collection1/
[junit4:junit4]   2> 12977 T422 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:65134/solr
[junit4:junit4]   2> 12977 T422 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 12977 T422 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C12 name=collection1 org.apache.solr.core.SolrCore@3f696b26 url=http://127.0.0.1:65134/solr/collection1 node=127.0.0.1:65134_solr
[junit4:junit4]   2> 12979 T517 C12 P65134 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 12979 T517 C12 P65134 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 12979 T422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 12979 T517 C12 P65134 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 12980 T422 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 12981 T422 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 12981 T517 C12 P65134 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:23255/solr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 12981 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 12982 T517 C12 P65134 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:65134/solr START replicas=[http://127.0.0.1:23255/solr/collection1/] nUpdates=100
[junit4:junit4]   2> 12982 T517 C12 P65134 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 12983 T517 C12 P65134 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 12983 T517 C12 P65134 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 12987 T517 C12 P65134 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 12990 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 12990 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:65134_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:65134/solr"}
[junit4:junit4]   2> ASYNC  NEW_CORE C13 name=collection1 org.apache.solr.core.SolrCore@77f4bff5 url=http://127.0.0.1:23255/solr/collection1 node=127.0.0.1:23255_solr
[junit4:junit4]   2> 12996 T478 C13 P23255 REQ /get {distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=0 
[junit4:junit4]   2> 13005 T422 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:17418/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@685ba5b6
[junit4:junit4]   2> 13006 T477 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 13006 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 13006 T493 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 13006 T515 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 13006 T422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13006 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 13006 T519 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:17418
[junit4:junit4]   2> 13007 T519 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:17418, initiating session
[junit4:junit4]   2> 13008 T424 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:28709
[junit4:junit4]   2> 13008 T424 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:28709
[junit4:junit4]   2> 13010 T426 oazs.NIOServerCnxn.finishSessionInit Established session 0x13adb63f4c60007 with negotiated timeout 10000 for client /140.211.11.196:28709
[junit4:junit4]   2> 13010 T519 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:17418, sessionid = 0x13adb63f4c60007, negotiated timeout = 10000
[junit4:junit4]   2> 13010 T520 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@685ba5b6 name:ZooKeeperConnection Watcher:127.0.0.1:17418/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13010 T422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13011 T422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 13015 T422 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 13016 T422 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 13024 T422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C14 name=collection1 org.apache.solr.core.SolrCore@4a5ae26e url=http://127.0.0.1:16546/solr/collection1 node=127.0.0.1:16546_solr
[junit4:junit4]   2> 13139 T446 C14 P16546 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16
[junit4:junit4]   2> 13200 T446 C14 P16546 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/control/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 13201 T446 C14 P16546 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 13232 T446 C14 P16546 /update {wt=javabin&version=2} {add=[1 (1417989854489214976)]} 0 125
[junit4:junit4]   2> 13233 T422 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:17418/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@2ee834b2
[junit4:junit4]   2> 13234 T521 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:17418
[junit4:junit4]   2> 13234 T422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13235 T521 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:17418, initiating session
[junit4:junit4]   2> 13235 T424 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:23838
[junit4:junit4]   2> 13235 T424 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:23838
[junit4:junit4]   2> 13236 T426 oazs.NIOServerCnxn.finishSessionInit Established session 0x13adb63f4c60008 with negotiated timeout 10000 for client /140.211.11.196:23838
[junit4:junit4]   2> 13236 T521 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:17418, sessionid = 0x13adb63f4c60008, negotiated timeout = 10000
[junit4:junit4]   2> 13236 T522 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ee834b2 name:ZooKeeperConnection Watcher:127.0.0.1:17418/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13237 T422 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13237 T422 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 13256 T463 C11 P65407 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty1/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 13256 T463 C11 P65407 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2>  C10_STATE=coll:collection1 core:collection1 props:{shard=shard1, roles=null, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:32669_solr, base_url=http://127.0.0.1:32669/solr}
[junit4:junit4]   2> 13307 T496 C10 P32669 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 13317 T496 C10 P32669 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 13323 T496 C10 P32669 REQ /update {distrib.from=http://127.0.0.1:65407/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=21 
[junit4:junit4]   2> 13324 T463 C11 P65407 /update {wt=javabin&version=2} {add=[1 (1417989854548983808)]} 0 77
[junit4:junit4]   2> 13332 T448 C14 P16546 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 13396 T448 C14 P16546 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/control/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/control/data/index,segFN=segments_2,generation=2,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 13397 T448 C14 P16546 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 13411 T448 C14 P16546 oass.SolrIndexSearcher.<init> Opening Searcher@bcbc73 main
[junit4:junit4]   2> 13412 T448 C14 P16546 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 13413 T450 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bcbc73 main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 13414 T448 C14 P16546 /update {waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 82
[junit4:junit4]   2> 13418 T465 C11 P65407 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 13480 T465 C11 P65407 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty1/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty1/index,segFN=segments_2,generation=2,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 13480 T465 C11 P65407 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 13493 T465 C11 P65407 oass.SolrIndexSearcher.<init> Opening Searcher@6075484f main
[junit4:junit4]   2> 13494 T465 C11 P65407 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 13495 T466 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6075484f main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 13496 T465 C11 P65407 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:32669/solr/collection1/, StdNode: http://127.0.0.1:23255/solr/collection1/, StdNode: http://127.0.0.1:65134/solr/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 13500 T497 C10 P32669 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 13500 T481 C13 P23255 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 13510 T481 C13 P23255 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty2/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 13512 T481 C13 P23255 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 13513 T481 C13 P23255 oass.SolrIndexSearcher.<init> Opening Searcher@d535d91 main
[junit4:junit4]   2> 13514 T481 C13 P23255 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2>  C12_STATE=coll:collection1 core:collection1 props:{shard=shard2, roles=null, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:65134_solr, base_url=http://127.0.0.1:65134/solr}
[junit4:junit4]   2> 13514 T511 C12 P65134 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 13515 T482 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d535d91 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 13516 T481 C13 P23255 /update {waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 16
[junit4:junit4]   2> 13519 T511 C12 P65134 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty4/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 13520 T511 C12 P65134 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 13521 T511 C12 P65134 oass.SolrIndexSearcher.<init> Opening Searcher@5012a15b main
[junit4:junit4]   2> 13522 T511 C12 P65134 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 13523 T516 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5012a15b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 13523 T511 C12 P65134 /update {waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 9
[junit4:junit4]   2> 13552 T497 C10 P32669 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index,segFN=segments_2,generation=2,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 13552 T497 C10 P32669 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 13569 T497 C10 P32669 oass.SolrIndexSearcher.<init> Opening Searcher@18c1c9eb main
[junit4:junit4]   2> 13570 T497 C10 P32669 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 13571 T498 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18c1c9eb main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 13571 T497 C10 P32669 /update {waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 72
[junit4:junit4]   2> 13572 T465 C11 P65407 /update {waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 154
[junit4:junit4]   2> 13573 T422 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):540
[junit4:junit4]   2> 13574 T422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 13768 T499 C10 P32669 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 13768 T499 C10 P32669 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 13768 T499 C10 P32669 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:65407/solr/collection1/. core=collection1
[junit4:junit4]   2> 13769 T499 C10 P32669 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 13790 T464 C11 P65407 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 13795 T464 C11 P65407 oass.SolrIndexSearcher.<init> Opening Searcher@671e6e1d realtime
[junit4:junit4]   2> 13795 T464 C11 P65407 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 13796 T464 C11 P65407 /update {waitSearcher=true&openSearcher=false&commit=true&commit_end_point=true&wt=javabin&softCommit=false&version=2} {commit=} 0 6
[junit4:junit4]   2> 13797 T499 C10 P32669 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 13797 T499 C10 P32669 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4]   2> 13826 T499 C10 P32669 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 13897 T465 C11 P65407 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 13898 T465 C11 P65407 REQ /replication {command=indexversion&wt=javabin&qt=/replication&version=2} status=0 QTime=69 
[junit4:junit4]   2> 13899 T499 C10 P32669 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 13899 T499 C10 P32669 oash.SnapPuller.fetchLatestIndex Slave's generation: 2
[junit4:junit4]   2> 13899 T499 C10 P32669 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 13904 T464 C11 P65407 REQ /replication {command=filelist&generation=2&wt=javabin&qt=/replication&version=2} status=0 QTime=2 
[junit4:junit4]   2> 13905 T499 C10 P32669 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 11
[junit4:junit4]   2> 13907 T499 C10 P32669 oash.SnapPuller.fetchLatestIndex Starting download to ./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index.20121107050145861 fullCopy=true
[junit4:junit4]   2> 13922 T463 C11 P65407 REQ /replication {file=_0.fnm&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=4 
[junit4:junit4]   2> 13943 T465 C11 P65407 REQ /replication {file=_0_nrm.cfe&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=1 
[junit4:junit4]   2> 13948 T464 C11 P65407 REQ /replication {file=_0.tim&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 13954 T463 C11 P65407 REQ /replication {file=_0.frq&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 13958 T465 C11 P65407 REQ /replication {file=segments_2&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 13963 T464 C11 P65407 REQ /replication {file=_0.fdx&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 13968 T463 C11 P65407 REQ /replication {file=_0_nrm.cfs&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 13972 T465 C11 P65407 REQ /replication {file=_0.si&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 13977 T463 C11 P65407 REQ /replication {file=_0.tip&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 13981 T464 C11 P65407 REQ /replication {file=_0.prx&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=0 
[junit4:junit4]   2> 13986 T465 C11 P65407 REQ /replication {file=_0.fdt&command=filecontent&checksum=true&generation=2&qt=/replication&wt=filestream} status=0 QTime=1 
[junit4:junit4]   2> 13987 T499 C10 P32669 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 13996 T499 C10 P32669 oash.SnapPuller.modifyIndexProps New index installed. Updating index properties... index=index.20121107050145861
[junit4:junit4]   2> 14009 T499 C10 P32669 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 14009 T499 C10 P32669 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 14009 T499 C10 P32669 oasu.DefaultSolrCoreState.newIndexWriter Rollback old IndexWriter... core=collection1
[junit4:junit4]   2> 14012 T499 C10 P32669 oasc.SolrCore.getNewIndexDir New index directory detected: old=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index/ new=./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index.20121107050145861
[junit4:junit4]   2> 14013 T499 C10 P32669 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index.20121107050145861 forceNew:true
[junit4:junit4]   2> 14021 T499 C10 P32669 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index.20121107050145861,segFN=segments_2,generation=2,filenames=[_0.fnm, _0_nrm.cfe, _0.tim, _0.frq, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.tip, _0.prx, _0.fdt]
[junit4:junit4]   2> 14021 T499 C10 P32669 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 14022 T499 C10 P32669 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 14025 T499 C10 P32669 oass.SolrIndexSearcher.<init> Opening Searcher@3c9271c0 main
[junit4:junit4]   2> 14027 T498 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3c9271c0 main{StandardDirectoryReader(segments_2:3:nrt _0(4.1):C1)}
[junit4:junit4]   2> 14027 T498 oasc.CachingDirectoryFactory.close Closing directory:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index
[junit4:junit4]   2> 14027 T498 oash.SnapPuller$2.onClose removing old index directory ./org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty3/index
[junit4:junit4]   2> 14035 T499 C10 P32669 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=false,waitSearcher=false,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 14036 T499 C10 P32669 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 14036 T499 C10 P32669 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 14036 T499 C10 P32669 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 14036 T499 C10 P32669 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 14038 T499 C10 P32669 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 14516 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14517 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:32669_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:32669/solr"}
[junit4:junit4]   2> 14538 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 14538 T477 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 14538 T522 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 14538 T515 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 14538 T493 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 14538 T520 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 14538 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 14576 T422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 15578 T422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16008 T517 C12 P65134 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 16008 T517 C12 P65134 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 16008 T517 C12 P65134 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:23255/solr/collection1/. core=collection1
[junit4:junit4]   2> 16008 T517 C12 P65134 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16032 T478 C13 P23255 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 16033 T478 C13 P23255 oass.SolrIndexSearcher.<init> Opening Searcher@4c1233d7 realtime
[junit4:junit4]   2> 16034 T478 C13 P23255 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 16034 T478 C13 P23255 /update {waitSearcher=true&openSearcher=false&commit=true&commit_end_point=true&wt=javabin&softCommit=false&version=2} {commit=} 0 2
[junit4:junit4]   2> 16035 T517 C12 P65134 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 16036 T517 C12 P65134 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 16038 T480 C13 P23255 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 16039 T480 C13 P23255 REQ /replication {command=indexversion&wt=javabin&qt=/replication&version=2} status=0 QTime=1 
[junit4:junit4]   2> 16042 T517 C12 P65134 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 16047 T517 C12 P65134 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty4/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.cloud.BasicDistributedZk2Test-1352300491961/jetty4/index,segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 16047 T517 C12 P65134 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 16052 T517 C12 P65134 oass.SolrIndexSearcher.<init> Opening Searcher@256b4372 main
[junit4:junit4]   2> 16052 T517 C12 P65134 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 16054 T516 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@256b4372 main{StandardDirectoryReader(segments_2:2)}
[junit4:junit4]   2> 16054 T517 C12 P65134 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 16054 T517 C12 P65134 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 16054 T517 C12 P65134 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16056 T517 C12 P65134 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 16549 T444 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16550 T444 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:65134_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:65134/solr"}
[junit4:junit4]   2> 16555 T477 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 16555 T443 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 16555 T461 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 16555 T522 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 16556 T515 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 16555 T520 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 16555 T493 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (5)
[junit4:junit4]   2> 16581 T422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 16605 T446 C14 P16546 REQ /select {fl=id,score&shard.url=127.0.0.1:16546/solr/collection1/&NOW=1352300508556&start=0&q=*:*&distrib=false&isShard=true&wt=javabin&fsv=true&rows=10&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 16632 T449 C14 P16546 REQ /select {shard.url=127.0.0.1:16546/solr/collection1/&NOW=1352300508556&q=*:*&ids=1&distrib=false&isShard=true&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 16634 T448 C14 P16546 REQ /select {wt=javabin&q=*:*&version=2} hits=1 status=0 QTime=34 
[junit4:junit4]   2> 16635 T422 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:17418/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@7e7f9d6b
[junit4:junit4]   2> 16636 T528 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:17418
[junit4:junit4]   2> 16636 T422 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16637 T528 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:17418,

[...truncated too long message...]

LeakControl.java:782)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]   2> 	        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
[junit4:junit4]   2> 	        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]   2> 	        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]   2> 	        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]   2> 	        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]   2> 	        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  28) Thread[id=70, name=qtp2035256218-70 Acceptor3 SelectChannelConnector@0.0.0.0:57105, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  29) Thread[id=130, name=TEST-FullSolrCloudDistribCmdsTest.testDistribSearch-seed#[FF514C3648EA9C60]-SendThread(localhost:49916), state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueArrayWrapper.interrupt(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueArrayWrapper.interrupt(KqueueArrayWrapper.java:206)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueSelectorImpl.wakeup(KqueueSelectorImpl.java:194)
[junit4:junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
[junit4:junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:215)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueSelectorImpl.doSelect(KqueueSelectorImpl.java:68)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4:junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1131)
[junit4:junit4]   2> 	  30) Thread[id=120, name=qtp20084428-120 Acceptor3 SelectChannelConnector@0.0.0.0:30577, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  31) Thread[id=119, name=qtp20084428-119 Acceptor2 SelectChannelConnector@0.0.0.0:30577, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  32) Thread[id=82, name=qtp491114805-82 Selector3, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueArrayWrapper.interrupt(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueArrayWrapper.interrupt(KqueueArrayWrapper.java:206)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueSelectorImpl.wakeup(KqueueSelectorImpl.java:194)
[junit4:junit4]   2> 	        at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
[junit4:junit4]   2> 	        at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:215)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueSelectorImpl.doSelect(KqueueSelectorImpl.java:68)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4:junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4:junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=da_DK, timezone=Chile/Continental
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=40,free=377561360,total=426049536
[junit4:junit4]   2> NOTE: All tests run in this JVM: [IndexSchemaRuntimeFieldTest, TestPropInjectDefaults, TestSolrJ, QueryParsingTest, TestQueryUtils, IndexSchemaTest, DocumentBuilderTest, BasicFunctionalityTest, FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.seed=FF514C3648EA9C60 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=da_DK -Dtests.timezone=Chile/Continental -Dtests.file.encoding=ISO8859-1
[junit4:junit4] ERROR   0.00s J1 | FullSolrCloudDistribCmdsTest (suite) <<<
[junit4:junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([FF514C3648EA9C60]:0)
[junit4:junit4] Completed on J1 in 7224.06s, 1 test, 2 errors <<< FAILURES!

[...truncated 162 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 970 tests, 3 suite-level errors, 1 error, 750 ignored (2 assumptions)

Total time: 139 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 837 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/837/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.AnalysisAfterCoreReloadTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.AnalysisAfterCoreReloadTest:     1) Thread[id=87, name=qtp440019538-87 Acceptor2 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=88, name=qtp440019538-88 Acceptor3 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=85, name=qtp440019538-85 Acceptor0 SelectChannelConnector@0.0.0.0:45607, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=86, name=qtp440019538-86 Acceptor1 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.AnalysisAfterCoreReloadTest: 
   1) Thread[id=87, name=qtp440019538-87 Acceptor2 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=88, name=qtp440019538-88 Acceptor3 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=85, name=qtp440019538-85 Acceptor0 SelectChannelConnector@0.0.0.0:45607, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=86, name=qtp440019538-86 Acceptor1 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([7D99B3B79A418E80]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.AnalysisAfterCoreReloadTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=87, name=qtp440019538-87 Acceptor2 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=88, name=qtp440019538-88 Acceptor3 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=85, name=qtp440019538-85 Acceptor0 SelectChannelConnector@0.0.0.0:45607, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=86, name=qtp440019538-86 Acceptor1 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=87, name=qtp440019538-87 Acceptor2 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=88, name=qtp440019538-88 Acceptor3 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=85, name=qtp440019538-85 Acceptor0 SelectChannelConnector@0.0.0.0:45607, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=86, name=qtp440019538-86 Acceptor1 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([7D99B3B79A418E80]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.request.TestRemoteStreaming

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming:     1) Thread[id=53, name=qtp1537376642-53 Acceptor0 SelectChannelConnector@0.0.0.0:49506, state=RUNNABLE, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=54, name=qtp1537376642-54 Acceptor1 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=55, name=qtp1537376642-55 Acceptor2 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=56, name=qtp1537376642-56 Acceptor3 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming: 
   1) Thread[id=53, name=qtp1537376642-53 Acceptor0 SelectChannelConnector@0.0.0.0:49506, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=54, name=qtp1537376642-54 Acceptor1 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=55, name=qtp1537376642-55 Acceptor2 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=56, name=qtp1537376642-56 Acceptor3 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([7D99B3B79A418E80]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.request.TestRemoteStreaming

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=53, name=qtp1537376642-53 Acceptor0 SelectChannelConnector@0.0.0.0:49506, state=RUNNABLE, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=54, name=qtp1537376642-54 Acceptor1 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=55, name=qtp1537376642-55 Acceptor2 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=56, name=qtp1537376642-56 Acceptor3 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=53, name=qtp1537376642-53 Acceptor0 SelectChannelConnector@0.0.0.0:49506, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=54, name=qtp1537376642-54 Acceptor1 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=55, name=qtp1537376642-55 Acceptor2 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=56, name=qtp1537376642-56 Acceptor3 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([7D99B3B79A418E80]:0)




Build Log:
[...truncated 7804 lines...]
[junit4:junit4] Suite: org.apache.solr.request.TestRemoteStreaming
[junit4:junit4]   2> 0 T46 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestRemoteStreaming-1352234153754
[junit4:junit4]   2> 1 T46 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 435 T46 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 724 T46 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:49506
[junit4:junit4]   2> 755 T46 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 755 T46 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: solr/
[junit4:junit4]   2> 756 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: 'solr/'
[junit4:junit4]   2> 788 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 788 T46 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 789 T46 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: solr/
[junit4:junit4]   2> 789 T46 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solr/solr.xml
[junit4:junit4]   2> 789 T46 oasc.CoreContainer.<init> New CoreContainer 968084630
[junit4:junit4]   2> 790 T46 oasc.CoreContainer$Initializer.initialize no solr.xml file found - using default
[junit4:junit4]   2> 790 T46 oasc.CoreContainer.load Loading CoreContainer using Solr Home: 'solr/'
[junit4:junit4]   2> 790 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/'
[junit4:junit4]   2> 810 T46 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 827 T46 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: solr/collection1
[junit4:junit4]   2> 828 T46 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/collection1/'
[junit4:junit4]   2> 902 T46 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 962 T46 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 965 T46 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 978 T46 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1490 T46 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1503 T46 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1508 T46 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1534 T46 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1541 T46 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1545 T46 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1552 T46 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1556 T46 oasc.SolrCore.<init> [collection1] Opening new SolrCore at solr/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-TestRemoteStreaming-1352234153754/
[junit4:junit4]   2> 1556 T46 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c2c9103
[junit4:junit4]   2> 1558 T46 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-TestRemoteStreaming-1352234153754/index/
[junit4:junit4]   2> 1558 T46 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-TestRemoteStreaming-1352234153754/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1559 T46 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-TestRemoteStreaming-1352234153754/index forceNew:false
[junit4:junit4]   2> 1561 T46 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1561 T46 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1562 T46 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 1563 T46 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1564 T46 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 1564 T46 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1565 T46 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 1566 T46 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1566 T46 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1567 T46 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1567 T46 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1568 T46 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1568 T46 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 1569 T46 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 1569 T46 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1570 T46 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 1570 T46 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 1571 T46 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 1571 T46 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1572 T46 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1572 T46 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1573 T46 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1573 T46 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1574 T46 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1574 T46 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1575 T46 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1575 T46 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 1576 T46 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 1582 T46 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1587 T46 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 1588 T46 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: []
[junit4:junit4]   2> 1591 T46 oass.SolrIndexSearcher.<init> Opening Searcher@78221c75 main
[junit4:junit4]   2> 1591 T46 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1592 T46 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1592 T46 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1593 T46 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 1640 T46 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 1829 T46 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1830 T46 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1830 T46 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1830 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1831 T46 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1831 T46 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1831 T46 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1832 T46 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1832 T46 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1832 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1848 T59 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@78221c75 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1851 T46 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1852 T46 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 1852 T46 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1852 T46 oas.SolrJettyTestBase.createJetty Jetty Assigned Port#49506
[junit4:junit4]   2> 1857 T46 oas.SolrTestCaseJ4.setUp ###Starting testMakeDeleteAllUrl
[junit4:junit4]   2> 1907 T46 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@52a0fbd6
[junit4:junit4]   2> 3094 T58 C7 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3094 T58 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3097 T58 C7 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1234]} 0 48
[junit4:junit4]   2> 3302 T60 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 3307 T60 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Lucene41_0.doc, _0_Lucene41_0.pos, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0.fdt, _0_Lucene41_0.tip]
[junit4:junit4]   2> 3307 T60 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 3309 T60 C7 oass.SolrIndexSearcher.<init> Opening Searcher@71c94ff3 main
[junit4:junit4]   2> 3310 T60 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3310 T60 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 3313 T59 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@71c94ff3 main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 3314 T60 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 13
[junit4:junit4]   2> 3394 T60 C7 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=1 status=0 QTime=49 
[junit4:junit4]   2> 3414 T58 C7 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 3415 T58 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 3416 T58 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Lucene41_0.doc, _0_Lucene41_0.pos, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0.fdt, _0_Lucene41_0.tip]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 3416 T58 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 3417 T58 C7 oass.SolrIndexSearcher.<init> Opening Searcher@39bbf341 main
[junit4:junit4]   2> 3418 T58 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3418 T58 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 3420 T59 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@39bbf341 main{StandardDirectoryReader(segments_3:4)}
[junit4:junit4]   2> 3421 T58 C7 UPDATE [collection1] webapp=/solr path=/update params={commit=true&stream.body=<delete><query>*:*</query></delete>} {deleteByQuery=*:*,commit=} 0 8
[junit4:junit4]   2> 3439 T60 C7 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3441 T46 oas.SolrTestCaseJ4.tearDown ###Ending testMakeDeleteAllUrl
[junit4:junit4]   2> 3449 T46 oas.SolrTestCaseJ4.setUp ###Starting testQtUpdateFails
[junit4:junit4]   2> 3454 T58 C7 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1234]} 0 1
[junit4:junit4]   2> 3458 T60 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 3461 T60 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_4,generation=4,filenames=[_1.fdx, _1_Lucene41_0.pos, _1_nrm.cfs, _1_Lucene41_0.doc, _1.fdt, _1.si, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _1.fnm, _1_nrm.cfe, segments_4]
[junit4:junit4]   2> 3462 T60 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 3464 T60 C7 oass.SolrIndexSearcher.<init> Opening Searcher@432b71eb main
[junit4:junit4]   2> 3464 T60 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3465 T60 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 3467 T59 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@432b71eb main{StandardDirectoryReader(segments_4:6 _1(4.1):C1)}
[junit4:junit4]   2> 3468 T60 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 10
[junit4:junit4]   2> 3472 T58 C7 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 3477 T46 oas.SolrTestCaseJ4.tearDown ###Ending testQtUpdateFails
[junit4:junit4]   2> 3485 T46 oas.SolrTestCaseJ4.setUp ###Starting testStreamUrl
[junit4:junit4]   2> 3490 T58 C7 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1234]} 0 2
[junit4:junit4]   2> 3494 T60 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 3499 T60 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_4,generation=4,filenames=[_1.fdx, _1_Lucene41_0.pos, _1_nrm.cfs, _1_Lucene41_0.doc, _1.fdt, _1.si, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _1.fnm, _1_nrm.cfe, segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_5,generation=5,filenames=[_2.si, _2_Lucene41_0.pos, _2_nrm.cfe, _2_Lucene41_0.tim, _2.fdt, _2_Lucene41_0.doc, _2_Lucene41_0.tip, _2_nrm.cfs, segments_5, _2.fdx, _2.fnm]
[junit4:junit4]   2> 3499 T60 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 5
[junit4:junit4]   2> 3502 T60 C7 oass.SolrIndexSearcher.<init> Opening Searcher@3a5dfc67 main
[junit4:junit4]   2> 3502 T60 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3503 T60 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 3505 T59 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a5dfc67 main{StandardDirectoryReader(segments_5:10 _2(4.1):C1)}
[junit4:junit4]   2> 3506 T60 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 12
[junit4:junit4]   2> 3510 T58 C7 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 3517 T58 C7 REQ [collection1] webapp=/solr path=/select params={wt=csv&q=*:*&fl=id} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 3568 T60 C7 REQ [collection1] webapp=/solr path=/debug/dump params={wt=xml&stream.url=http://127.0.0.1:49506/solr/select?q%3D*:*%26fl%3Did%26wt%3Dcsv} status=0 QTime=48 
[junit4:junit4]   2> 3569 T46 oas.SolrTestCaseJ4.tearDown ###Ending testStreamUrl
[junit4:junit4]   2> 3578 T46 oas.SolrTestCaseJ4.setUp ###Starting testNoUrlAccess
[junit4:junit4]   2> 3583 T62 C7 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1234]} 0 2
[junit4:junit4]   2> 3587 T60 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 3642 T60 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_5,generation=5,filenames=[_2.si, _2_Lucene41_0.pos, _2_nrm.cfe, _2_Lucene41_0.tim, _2.fdt, _2_Lucene41_0.doc, _2_Lucene41_0.tip, _2_nrm.cfs, segments_5, _2.fdx, _2.fnm]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d0cface lockFactory=org.apache.lucene.store.NativeFSLockFactory@42b1b290),segFN=segments_6,generation=6,filenames=[_3.fdx, _3_nrm.cfe, _3_nrm.cfs, _3_Lucene41_0.doc, _3_Lucene41_0.pos, _3_Lucene41_0.tip, _3.si, _3.fdt, _3_Lucene41_0.tim, _3.fnm, segments_6]
[junit4:junit4]   2> 3643 T60 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 6
[junit4:junit4]   2> 3646 T60 C7 oass.SolrIndexSearcher.<init> Opening Searcher@4123a08e main
[junit4:junit4]   2> 3646 T60 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3647 T60 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 3649 T59 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4123a08e main{StandardDirectoryReader(segments_6:14 _3(4.1):C1)}
[junit4:junit4]   2> 3650 T60 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 63
[junit4:junit4]   2> 3656 T58 C7 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 3667 T62 C7 REQ [collection1] webapp=/solr path=/select params={wt=javabin&stream.url=http://127.0.0.1:49506/solr/update?commit%3Dtrue%26stream.body%3D%253Cdelete%253E%253Cquery%253E*%253A*%253C%252Fquery%253E%253C%252Fdelete%253E&q=*:*&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 3671 T60 C7 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 3672 T46 oas.SolrTestCaseJ4.tearDown ###Ending testNoUrlAccess
[junit4:junit4]   2> 3719 T46 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=968084630
[junit4:junit4]   2> 3719 T46 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@52a0fbd6
[junit4:junit4]   2> 3733 T46 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=5,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=4,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 3733 T46 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 3734 T46 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 3734 T46 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 3736 T46 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 3741 T46 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 3843 T46 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 3843 T46 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 3858 T45 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 23953 T45 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming: 
[junit4:junit4]   2> 	   1) Thread[id=53, name=qtp1537376642-53 Acceptor0 SelectChannelConnector@0.0.0.0:49506, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=54, name=qtp1537376642-54 Acceptor1 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=55, name=qtp1537376642-55 Acceptor2 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=56, name=qtp1537376642-56 Acceptor3 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 23954 T45 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=53, name=qtp1537376642-53 Acceptor0 SelectChannelConnector@0.0.0.0:49506, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	   2) Thread[id=54, name=qtp1537376642-54 Acceptor1 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	   3) Thread[id=55, name=qtp1537376642-55 Acceptor2 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	   4) Thread[id=56, name=qtp1537376642-56 Acceptor3 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 26997 T45 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=53, name=qtp1537376642-53 Acceptor0 SelectChannelConnector@0.0.0.0:49506, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=54, name=qtp1537376642-54 Acceptor1 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=55, name=qtp1537376642-55 Acceptor2 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=56, name=qtp1537376642-56 Acceptor3 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Compressing(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=93)), sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_SY, timezone=Pacific/Marquesas
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=400782528,total=473104384
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestArbitraryIndexDir, TestMultiCoreConfBootstrap, PrimitiveFieldTypeTest, TestSort, TestQueryUtils, SuggesterTSTTest, DirectUpdateHandlerTest, TestRemoteStreaming]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRemoteStreaming -Dtests.seed=7D99B3B79A418E80 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_SY -Dtests.timezone=Pacific/Marquesas -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR   0.00s J1 | TestRemoteStreaming (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming: 
[junit4:junit4]    >    1) Thread[id=53, name=qtp1537376642-53 Acceptor0 SelectChannelConnector@0.0.0.0:49506, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=54, name=qtp1537376642-54 Acceptor1 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=55, name=qtp1537376642-55 Acceptor2 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=56, name=qtp1537376642-56 Acceptor3 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([7D99B3B79A418E80]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=53, name=qtp1537376642-53 Acceptor0 SelectChannelConnector@0.0.0.0:49506, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=54, name=qtp1537376642-54 Acceptor1 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=55, name=qtp1537376642-55 Acceptor2 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=56, name=qtp1537376642-56 Acceptor3 SelectChannelConnector@0.0.0.0:49506, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([7D99B3B79A418E80]:0)
[junit4:junit4] Completed on J1 in 27.18s, 4 tests, 2 errors <<< FAILURES!

[...truncated 181 lines...]
[junit4:junit4] Suite: org.apache.solr.AnalysisAfterCoreReloadTest
[junit4:junit4]   2> 988 T77 oasu.AbstractSolrTestCase.setUp ####SETUP_START testStopwordsAfterCoreReload
[junit4:junit4]   2> 990 T77 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/'
[junit4:junit4]   2> 991 T77 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solr-test-home-21201041200811743/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 991 T77 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solr-test-home-21201041200811743/collection1/lib/README' to classloader
[junit4:junit4]   2> 1051 T77 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1096 T77 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1097 T77 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1101 T77 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1504 T77 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1512 T77 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1515 T77 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1524 T77 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1528 T77 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1531 T77 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1532 T77 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1532 T77 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1533 T77 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743
[junit4:junit4]   2> 1533 T77 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/'
[junit4:junit4]   2> 1540 T77 oasc.CoreContainer.<init> New CoreContainer 1300817040
[junit4:junit4]   2> 1541 T77 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.AnalysisAfterCoreReloadTest-1352234157279/
[junit4:junit4]   2> 1541 T77 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fde891b
[junit4:junit4]   2> 1542 T77 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.AnalysisAfterCoreReloadTest-1352234157279/index/
[junit4:junit4]   2> 1542 T77 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.AnalysisAfterCoreReloadTest-1352234157279/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1544 T77 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.AnalysisAfterCoreReloadTest-1352234157279/index forceNew:false
[junit4:junit4]   2> 1546 T77 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.SimpleFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/index6666266663tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@63c5d81c),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1547 T77 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1547 T77 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 1549 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1549 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 1550 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1550 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 1551 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1551 T77 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1551 T77 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1552 T77 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1552 T77 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1553 T77 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 1553 T77 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 1554 T77 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1554 T77 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 1554 T77 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 1555 T77 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 1555 T77 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1555 T77 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1556 T77 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1556 T77 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1557 T77 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1557 T77 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1558 T77 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1558 T77 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1559 T77 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 1559 T77 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 1564 T77 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1567 T77 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 1568 T77 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMA-MINIMAL.XML, SCHEMA-REPLICATION2.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, SOLRCONFIG-MERGEPOLICY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, CURRENCY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, ELEVATE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, SCHEMA-REPLICATION1.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, DA_UTF8.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SOLRCONFIG-ENABLEPLUGIN.XML, HYPHENATION.DTD, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, SCHEMA-XINCLUDE.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SOLRCONFIG-TERMINDEX.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-ELEVATE.XML, SCHEMA-FOLDING.XML, STOPWORDS.TXT, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, BAD-MP-SOLRCONFIG.XML, FRENCHARTICLES.TXT, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, XSLT, SOLRCONFIG-NATIVELOCK.XML, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-ALTDIRECTORY.XML, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, SOLRCONFIG_PERF.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 1571 T77 oass.SolrIndexSearcher.<init> Opening Searcher@2dbe1f3e main
[junit4:junit4]   2> 1572 T77 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1572 T77 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1573 T77 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1573 T77 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 1592 T77 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 1626 T77 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1627 T77 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1627 T77 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1627 T77 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1627 T77 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1628 T77 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1628 T77 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1628 T77 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1629 T77 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1629 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1642 T78 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2dbe1f3e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1645 T77 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1650 T77 oasu.AbstractSolrTestCase.setUp ####SETUP_END testStopwordsAfterCoreReload
[junit4:junit4]   2> 1985 T77 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2215 T77 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:45607
[junit4:junit4]   2> 2240 T77 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2240 T77 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743
[junit4:junit4]   2> 2241 T77 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/'
[junit4:junit4]   2> 2267 T77 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2268 T77 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2268 T77 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743
[junit4:junit4]   2> 2269 T77 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/solr.xml
[junit4:junit4]   2> 2269 T77 oasc.CoreContainer.<init> New CoreContainer 1047960954
[junit4:junit4]   2> 2269 T77 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/'
[junit4:junit4]   2> 2270 T77 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/'
[junit4:junit4]   2> 2287 T77 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2300 T77 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1
[junit4:junit4]   2> 2301 T77 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/'
[junit4:junit4]   2> 2302 T77 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solr-test-home-21201041200811743/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2302 T77 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solr-test-home-21201041200811743/collection1/lib/README' to classloader
[junit4:junit4]   2> 2340 T77 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2385 T77 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2386 T77 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2392 T77 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2793 T77 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2802 T77 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2804 T77 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2815 T77 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2818 T77 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2821 T77 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2822 T77 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2823 T77 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/data/
[junit4:junit4]   2> 2824 T77 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fde891b
[junit4:junit4]   2> 2825 T77 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/data/index/
[junit4:junit4]   2> 2825 T77 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2826 T77 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/data/index forceNew:false
[junit4:junit4]   2> 2827 T77 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@697a9f24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a75d65c),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2828 T77 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2828 T77 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 2830 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2831 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 2831 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2832 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 2832 T77 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2833 T77 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2833 T77 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2834 T77 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2834 T77 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2835 T77 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 2835 T77 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 2836 T77 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2836 T77 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2837 T77 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2837 T77 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2838 T77 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2838 T77 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2839 T77 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2839 T77 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2840 T77 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2840 T77 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2841 T77 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2842 T77 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2842 T77 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 2843 T77 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 2847 T77 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2851 T77 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 2852 T77 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMA-MINIMAL.XML, SCHEMA-REPLICATION2.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, SOLRCONFIG-MERGEPOLICY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, CURRENCY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, ELEVATE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, SCHEMA-REPLICATION1.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, DA_UTF8.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SOLRCONFIG-ENABLEPLUGIN.XML, HYPHENATION.DTD, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, SCHEMA-XINCLUDE.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SOLRCONFIG-TERMINDEX.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-ELEVATE.XML, SCHEMA-FOLDING.XML, STOPWORDS.TXT, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, BAD-MP-SOLRCONFIG.XML, FRENCHARTICLES.TXT, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, XSLT, SOLRCONFIG-NATIVELOCK.XML, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-ALTDIRECTORY.XML, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, SOLRCONFIG_PERF.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 2854 T77 oass.SolrIndexSearcher.<init> Opening Searcher@4e2c390c main
[junit4:junit4]   2> 2855 T77 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 2855 T77 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2856 T77 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2856 T77 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 2858 T77 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2868 T77 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2868 T77 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2869 T77 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2869 T77 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2869 T77 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2870 T77 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2870 T77 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2870 T77 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2871 T77 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2871 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2887 T93 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4e2c390c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2890 T77 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2891 T77 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2891 T77 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2963 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@4ffb06c9
[junit4:junit4]   2> 3956 T91 C4 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@697a9f24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a75d65c),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3957 T91 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3959 T91 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 4022 T91 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@697a9f24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a75d65c),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@697a9f24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a75d65c),segFN=segments_2,generation=2,filenames=[_0_Asserting_0.tim, _0_Asserting_0.tip, _0_MockFixedIntBlock_0.tib, _0_MockFixedIntBlock_0.pyl, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.tii, _0.si, _0_Asserting_0.doc, _0.fnm, _0_MockFixedIntBlock_0.pos, _0_nrm.cfs, _0_MockRandom_0.skp, _0_MockFixedIntBlock_0.frq, _0_MockRandom_0.tip, _0_MockRandom_0.sd, _0_MockRandom_0.frq, _0_MockFixedIntBlock_0.doc, _0_MockRandom_0.doc, _0_nrm.cfe, _0_MockRandom_0.pos, segments_2, _0.fdx, _0_Asserting_0.pos, _0_MockRandom_0.pyl, _0_MockRandom_0.tim, _0.fdt]
[junit4:junit4]   2> 4023 T91 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 4061 T91 C4 oass.SolrIndexSearcher.<init> Opening Searcher@5d23fab3 main
[junit4:junit4]   2> 4061 T91 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4062 T91 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 4064 T93 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5d23fab3 main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 4065 T91 C4 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {add=[42],commit=} 0 134
[junit4:junit4]   2> 4231 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4342 T92 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=teststop:terma&version=2} hits=1 status=0 QTime=59 
[junit4:junit4]   2> 4345 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4360 T91 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=teststop:stopworda&version=2} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 4361 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4377 T90 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=teststop:stopwordb&version=2} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 4378 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4393 T91 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=teststop:stopwordc&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 4395 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4423 T90 oasc.CoreContainer.reload Reloading SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1
[junit4:junit4]   2> 4424 T90 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/'
[junit4:junit4]   2> 4425 T90 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solr-test-home-21201041200811743/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 4426 T90 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solr-test-home-21201041200811743/collection1/lib/README' to classloader
[junit4:junit4]   2> 4506 T90 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 4572 T90 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 4573 T90 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 4581 T90 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 5140 T90 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5148 T90 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5150 T90 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5160 T90 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5163 T90 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5166 T90 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5167 T90 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5168 T90 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/data/
[junit4:junit4]   2> 5169 T90 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fde891b
[junit4:junit4]   2> 5170 T90 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solr-test-home-21201041200811743/collection1/data/index/
[junit4:junit4]   2> 5170 T90 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 5172 T90 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5173 T90 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 5173 T90 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 5173 T90 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 5174 T90 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 5174 T90 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 5175 T90 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 5175 T90 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5175 T90 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5176 T90 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 5176 T90 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 5177 T90 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5177 T90 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 5177 T90 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 5178 T90 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 5178 T90 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5179 T90 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5179 T90 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5180 T90 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5180 T90 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5181 T90 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5181 T90 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5181 T90 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5182 T90 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 5182 T90 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 5187 T90 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5188 T90 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 5190 T90 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMA-MINIMAL.XML, SCHEMA-REPLICATION2.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, SOLRCONFIG-MERGEPOLICY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, CURRENCY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, ELEVATE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, SCHEMA-REPLICATION1.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, DA_UTF8.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SOLRCONFIG-ENABLEPLUGIN.XML, HYPHENATION.DTD, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, SCHEMA-XINCLUDE.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SOLRCONFIG-TERMINDEX.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-ELEVATE.XML, SCHEMA-FOLDING.XML, STOPWORDS.TXT, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, BAD-MP-SOLRCONFIG.XML, FRENCHARTICLES.TXT, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, XSLT, SOLRCONFIG-NATIVELOCK.XML, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-ALTDIRECTORY.XML, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, SOLRCONFIG_PERF.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 5201 T90 oass.SolrIndexSearcher.<init> Opening Searcher@414d0e66 main
[junit4:junit4]   2> 5201 T90 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5202 T90 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5202 T90 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5202 T90 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 5204 T90 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 5218 T95 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@414d0e66 main{StandardDirectoryReader(segments_2:3:nrt _0(4.1):C1)}
[junit4:junit4]   2> 5221 T90 oasc.CoreContainer.register replacing core: collection1
[junit4:junit4]   2> 5222 T90 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4ffb06c9
[junit4:junit4]   2> 5226 T90 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 5227 T90 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 5229 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@427dac02
[junit4:junit4]   2> 5248 T94 C5 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5296 T94 C5 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@697a9f24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a75d65c),segFN=segments_2,generation=2,filenames=[_0_Asserting_0.tim, _0_Asserting_0.tip, _0_MockFixedIntBlock_0.tib, _0_MockFixedIntBlock_0.pyl, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.tii, _0.si, _0_Asserting_0.doc, _0.fnm, _0_MockFixedIntBlock_0.pos, _0_nrm.cfs, _0_MockRandom_0.skp, _0_MockFixedIntBlock_0.frq, _0_MockRandom_0.tip, _0_MockRandom_0.sd, _0_MockRandom_0.frq, _0_MockFixedIntBlock_0.doc, _0_MockRandom_0.doc, _0_nrm.cfe, _0_MockRandom_0.pos, segments_2, _0.fdx, _0_Asserting_0.pos, _0_MockRandom_0.pyl, _0_MockRandom_0.tim, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@697a9f24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a75d65c),segFN=segments_3,generation=3,filenames=[_1_nrm.cfs, _1_MockFixedIntBlock_0.tii, _1_Asserting_0.pos, _1_MockFixedIntBlock_0.frq, _1_MockFixedIntBlock_0.doc, _1_Asserting_0.doc, _1_MockRandom_0.tip, _1_MockFixedIntBlock_0.pyl, _1_MockRandom_0.sd, _1_MockRandom_0.tim, _1.fnm, _1_MockRandom_0.pos, _1_nrm.cfe, _1_Asserting_0.tim, _1.fdx, _1_MockFixedIntBlock_0.skp, _1.fdt, _1_Asserting_0.tip, _1.si, _1_MockFixedIntBlock_0.tib, _1_MockRandom_0.doc, _1_MockFixedIntBlock_0.pos, segments_3]
[junit4:junit4]   2> 5297 T94 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 5299 T94 C5 oass.SolrIndexSearcher.<init> Opening Searcher@4a05e593 main
[junit4:junit4]   2> 5300 T94 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5300 T94 C5 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5301 T95 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a05e593 main{StandardDirectoryReader(segments_3:7:nrt _1(4.1):C1)}
[junit4:junit4]   2> 5302 T94 C5 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {add=[42],commit=} 0 56
[junit4:junit4]   2> 5303 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 5314 T92 C5 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=teststop:terma&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 5315 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 5324 T91 C5 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=teststop:stopworda&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 5325 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 5335 T90 C5 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=teststop:stopwordb&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 5336 T77 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 5345 T92 C5 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=teststop:stopwordc&version=2} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 5390 T77 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1047960954
[junit4:junit4]   2> 5391 T77 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@427dac02
[junit4:junit4]   2> 5399 T77 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 5399 T77 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 5399 T77 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 5400 T77 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 5401 T77 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 5408 T77 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 5509 T77 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 5556 T77 oasu.AbstractSolrTestCase.tearDown ####TEARDOWN_START testStopwordsAfterCoreReload
[junit4:junit4]   2> 5557 T77 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1300817040
[junit4:junit4]   2> 5557 T77 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5655d1b4
[junit4:junit4]   2> 5559 T77 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 5559 T77 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 5559 T77 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 5559 T77 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 5560 T77 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 5573 T76 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 25669 T76 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 4 threads leaked from SUITE scope at org.apache.solr.AnalysisAfterCoreReloadTest: 
[junit4:junit4]   2> 	   1) Thread[id=87, name=qtp440019538-87 Acceptor2 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=88, name=qtp440019538-88 Acceptor3 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=85, name=qtp440019538-85 Acceptor0 SelectChannelConnector@0.0.0.0:45607, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=86, name=qtp440019538-86 Acceptor1 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 25670 T76 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=87, name=qtp440019538-87 Acceptor2 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	   2) Thread[id=88, name=qtp440019538-88 Acceptor3 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	   3) Thread[id=85, name=qtp440019538-85 Acceptor0 SelectChannelConnector@0.0.0.0:45607, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	   4) Thread[id=86, name=qtp440019538-86 Acceptor1 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 28714 T76 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=87, name=qtp440019538-87 Acceptor2 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=88, name=qtp440019538-88 Acceptor3 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=85, name=qtp440019538-85 Acceptor0 SelectChannelConnector@0.0.0.0:45607, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=86, name=qtp440019538-86 Acceptor1 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {timestamp=MockFixedIntBlock(blockSize=1306), range_facet_l=MockFixedIntBlock(blockSize=1306), id=PostingsFormat(name=Asserting), range_facet_sl=PostingsFormat(name=Asserting), range_facet_si=MockFixedIntBlock(blockSize=1306), teststop=MockFixedIntBlock(blockSize=1306), multiDefault=MockFixedIntBlock(blockSize=1306), intDefault=PostingsFormat(name=MockRandom)}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=hu_HU, timezone=CTT
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=328888560,total=474939392
[junit4:junit4]   2> NOTE: All tests run in this JVM: [IndexBasedSpellCheckerTest, CoreContainerCoreInitFailuresTest, TestCoreContainer, TestLFUCache, ClusterStateUpdateTest, MoreLikeThisHandlerTest, ResponseLogComponentTest, TestSearchPerf, SolrRequestParserTest, PathHierarchyTokenizerFactoryTest, SearchHandlerTest, AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AnalysisAfterCoreReloadTest -Dtests.seed=7D99B3B79A418E80 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=hu_HU -Dtests.timezone=CTT -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR   0.00s J0 | AnalysisAfterCoreReloadTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.AnalysisAfterCoreReloadTest: 
[junit4:junit4]    >    1) Thread[id=87, name=qtp440019538-87 Acceptor2 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=88, name=qtp440019538-88 Acceptor3 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=85, name=qtp440019538-85 Acceptor0 SelectChannelConnector@0.0.0.0:45607, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=86, name=qtp440019538-86 Acceptor1 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([7D99B3B79A418E80]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=87, name=qtp440019538-87 Acceptor2 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=88, name=qtp440019538-88 Acceptor3 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=85, name=qtp440019538-85 Acceptor0 SelectChannelConnector@0.0.0.0:45607, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=86, name=qtp440019538-86 Acceptor1 SelectChannelConnector@0.0.0.0:45607, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([7D99B3B79A418E80]:0)
[junit4:junit4] Completed on J0 in 28.01s, 1 test, 2 errors <<< FAILURES!

[...truncated 461 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 833 ignored

Total time: 17 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 836 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/836/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([38DF40773F94B8B2]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedSpellCheckComponentTest

Error Message:
36 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedSpellCheckComponentTest:     1) Thread[id=76, name=qtp2127994111-76 Acceptor1 SelectChannelConnector@0.0.0.0:64428, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=89, name=qtp1240176342-89 Acceptor0 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=135, name=qtp1340908330-135 Acceptor2 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=181, name=qtp939597125-181 Acceptor3 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=154, name=qtp1925964578-154 Acceptor1 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=91, name=qtp1240176342-91 Acceptor2 SelectChannelConnector@0.0.0.0:58229, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=178, name=qtp939597125-178 Acceptor0 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=195, name=qtp223292927-195 Acceptor2 SelectChannelConnector@0.0.0.0:42679, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=120, name=qtp1360890360-120 Acceptor1 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=106, name=qtp1547748565-106 Acceptor1 SelectChannelConnector@0.0.0.0:63574, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=166, name=qtp103469457-166 Acceptor2 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=152, name=qtp1925964578-152 Acceptor0 SelectChannelConnector@0.0.0.0:18757, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=122, name=qtp1360890360-122 Acceptor3 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=77, name=qtp2127994111-77 Acceptor2 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=90, name=qtp1240176342-90 Acceptor1 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=155, name=qtp1925964578-155 Acceptor2 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=164, name=qtp103469457-164 Acceptor0 SelectChannelConnector@0.0.0.0:38161, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=121, name=qtp1360890360-121 Acceptor2 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=134, name=qtp1340908330-134 Acceptor1 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=119, name=qtp1360890360-119 Acceptor0 SelectChannelConnector@0.0.0.0:50874, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=136, name=qtp1340908330-136 Acceptor3 SelectChannelConnector@0.0.0.0:49071, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=107, name=qtp1547748565-107 Acceptor2 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=108, name=qtp1547748565-108 Acceptor3 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=133, name=qtp1340908330-133 Acceptor0 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=92, name=qtp1240176342-92 Acceptor3 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=196, name=qtp223292927-196 Acceptor3 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=78, name=qtp2127994111-78 Acceptor3 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=167, name=qtp103469457-167 Acceptor3 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=153, name=qtp1925964578-153 Acceptor3 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=165, name=qtp103469457-165 Acceptor1 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=180, name=qtp939597125-180 Acceptor2 SelectChannelConnector@0.0.0.0:59174, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=194, name=qtp223292927-194 Acceptor1 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=75, name=qtp2127994111-75 Acceptor0 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=105, name=qtp1547748565-105 Acceptor0 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=193, name=qtp223292927-193 Acceptor0 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=179, name=qtp939597125-179 Acceptor1 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 36 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedSpellCheckComponentTest: 
   1) Thread[id=76, name=qtp2127994111-76 Acceptor1 SelectChannelConnector@0.0.0.0:64428, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=89, name=qtp1240176342-89 Acceptor0 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=135, name=qtp1340908330-135 Acceptor2 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=181, name=qtp939597125-181 Acceptor3 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=154, name=qtp1925964578-154 Acceptor1 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=91, name=qtp1240176342-91 Acceptor2 SelectChannelConnector@0.0.0.0:58229, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=178, name=qtp939597125-178 Acceptor0 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=195, name=qtp223292927-195 Acceptor2 SelectChannelConnector@0.0.0.0:42679, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=120, name=qtp1360890360-120 Acceptor1 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=106, name=qtp1547748565-106 Acceptor1 SelectChannelConnector@0.0.0.0:63574, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=166, name=qtp103469457-166 Acceptor2 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=152, name=qtp1925964578-152 Acceptor0 SelectChannelConnector@0.0.0.0:18757, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=122, name=qtp1360890360-122 Acceptor3 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=77, name=qtp2127994111-77 Acceptor2 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=90, name=qtp1240176342-90 Acceptor1 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=155, name=qtp1925964578-155 Acceptor2 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=164, name=qtp103469457-164 Acceptor0 SelectChannelConnector@0.0.0.0:38161, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=121, name=qtp1360890360-121 Acceptor2 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=134, name=qtp1340908330-134 Acceptor1 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=119, name=qtp1360890360-119 Acceptor0 SelectChannelConnector@0.0.0.0:50874, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=136, name=qtp1340908330-136 Acceptor3 SelectChannelConnector@0.0.0.0:49071, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=107, name=qtp1547748565-107 Acceptor2 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=108, name=qtp1547748565-108 Acceptor3 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=133, name=qtp1340908330-133 Acceptor0 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=92, name=qtp1240176342-92 Acceptor3 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=196, name=qtp223292927-196 Acceptor3 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=78, name=qtp2127994111-78 Acceptor3 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=167, name=qtp103469457-167 Acceptor3 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=153, name=qtp1925964578-153 Acceptor3 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=165, name=qtp103469457-165 Acceptor1 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=180, name=qtp939597125-180 Acceptor2 SelectChannelConnector@0.0.0.0:59174, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=194, name=qtp223292927-194 Acceptor1 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=75, name=qtp2127994111-75 Acceptor0 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=105, name=qtp1547748565-105 Acceptor0 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=193, name=qtp223292927-193 Acceptor0 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=179, name=qtp939597125-179 Acceptor1 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([38DF40773F94B8B2]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedSpellCheckComponentTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=76, name=qtp2127994111-76 Acceptor1 SelectChannelConnector@0.0.0.0:64428, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=89, name=qtp1240176342-89 Acceptor0 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=135, name=qtp1340908330-135 Acceptor2 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=181, name=qtp939597125-181 Acceptor3 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=154, name=qtp1925964578-154 Acceptor1 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=91, name=qtp1240176342-91 Acceptor2 SelectChannelConnector@0.0.0.0:58229, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=178, name=qtp939597125-178 Acceptor0 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=195, name=qtp223292927-195 Acceptor2 SelectChannelConnector@0.0.0.0:42679, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=120, name=qtp1360890360-120 Acceptor1 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=106, name=qtp1547748565-106 Acceptor1 SelectChannelConnector@0.0.0.0:63574, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=166, name=qtp103469457-166 Acceptor2 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=152, name=qtp1925964578-152 Acceptor0 SelectChannelConnector@0.0.0.0:18757, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=77, name=qtp2127994111-77 Acceptor2 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=122, name=qtp1360890360-122 Acceptor3 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=90, name=qtp1240176342-90 Acceptor1 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=155, name=qtp1925964578-155 Acceptor2 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=164, name=qtp103469457-164 Acceptor0 SelectChannelConnector@0.0.0.0:38161, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=121, name=qtp1360890360-121 Acceptor2 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=134, name=qtp1340908330-134 Acceptor1 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=119, name=qtp1360890360-119 Acceptor0 SelectChannelConnector@0.0.0.0:50874, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=136, name=qtp1340908330-136 Acceptor3 SelectChannelConnector@0.0.0.0:49071, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=107, name=qtp1547748565-107 Acceptor2 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=108, name=qtp1547748565-108 Acceptor3 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=133, name=qtp1340908330-133 Acceptor0 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=92, name=qtp1240176342-92 Acceptor3 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=196, name=qtp223292927-196 Acceptor3 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=78, name=qtp2127994111-78 Acceptor3 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=167, name=qtp103469457-167 Acceptor3 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=153, name=qtp1925964578-153 Acceptor3 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=165, name=qtp103469457-165 Acceptor1 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=180, name=qtp939597125-180 Acceptor2 SelectChannelConnector@0.0.0.0:59174, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=194, name=qtp223292927-194 Acceptor1 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=75, name=qtp2127994111-75 Acceptor0 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=105, name=qtp1547748565-105 Acceptor0 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=193, name=qtp223292927-193 Acceptor0 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=179, name=qtp939597125-179 Acceptor1 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=76, name=qtp2127994111-76 Acceptor1 SelectChannelConnector@0.0.0.0:64428, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=89, name=qtp1240176342-89 Acceptor0 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=135, name=qtp1340908330-135 Acceptor2 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=181, name=qtp939597125-181 Acceptor3 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=154, name=qtp1925964578-154 Acceptor1 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=91, name=qtp1240176342-91 Acceptor2 SelectChannelConnector@0.0.0.0:58229, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=178, name=qtp939597125-178 Acceptor0 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=195, name=qtp223292927-195 Acceptor2 SelectChannelConnector@0.0.0.0:42679, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=120, name=qtp1360890360-120 Acceptor1 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=106, name=qtp1547748565-106 Acceptor1 SelectChannelConnector@0.0.0.0:63574, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=166, name=qtp103469457-166 Acceptor2 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=152, name=qtp1925964578-152 Acceptor0 SelectChannelConnector@0.0.0.0:18757, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=77, name=qtp2127994111-77 Acceptor2 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=122, name=qtp1360890360-122 Acceptor3 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=90, name=qtp1240176342-90 Acceptor1 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=155, name=qtp1925964578-155 Acceptor2 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=164, name=qtp103469457-164 Acceptor0 SelectChannelConnector@0.0.0.0:38161, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=121, name=qtp1360890360-121 Acceptor2 SelectChannelConnector@0.0.0.0:50874, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=134, name=qtp1340908330-134 Acceptor1 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=119, name=qtp1360890360-119 Acceptor0 SelectChannelConnector@0.0.0.0:50874, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=136, name=qtp1340908330-136 Acceptor3 SelectChannelConnector@0.0.0.0:49071, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=107, name=qtp1547748565-107 Acceptor2 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=108, name=qtp1547748565-108 Acceptor3 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=133, name=qtp1340908330-133 Acceptor0 SelectChannelConnector@0.0.0.0:49071, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=92, name=qtp1240176342-92 Acceptor3 SelectChannelConnector@0.0.0.0:58229, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=196, name=qtp223292927-196 Acceptor3 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=78, name=qtp2127994111-78 Acceptor3 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=167, name=qtp103469457-167 Acceptor3 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=153, name=qtp1925964578-153 Acceptor3 SelectChannelConnector@0.0.0.0:18757, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=165, name=qtp103469457-165 Acceptor1 SelectChannelConnector@0.0.0.0:38161, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=180, name=qtp939597125-180 Acceptor2 SelectChannelConnector@0.0.0.0:59174, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=194, name=qtp223292927-194 Acceptor1 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=75, name=qtp2127994111-75 Acceptor0 SelectChannelConnector@0.0.0.0:64428, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=105, name=qtp1547748565-105 Acceptor0 SelectChannelConnector@0.0.0.0:63574, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=193, name=qtp223292927-193 Acceptor0 SelectChannelConnector@0.0.0.0:42679, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=179, name=qtp939597125-179 Acceptor1 SelectChannelConnector@0.0.0.0:59174, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([38DF40773F94B8B2]:0)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([38DF40773F94B8B2]:0)




Build Log:
[...truncated 7726 lines...]
[junit4:junit4] Suite: org.apache.solr.handler.component.DistributedSpellCheckComponentTest
[junit4:junit4]   2> 6 T68 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 660 T68 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1097 T68 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:64428
[junit4:junit4]   2> 1143 T68 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1143 T68 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1144 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1190 T68 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1191 T68 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1192 T68 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1192 T68 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1193 T68 oasc.CoreContainer.<init> New CoreContainer 703213836
[junit4:junit4]   2> 1194 T68 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1194 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1243 T68 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1315 T68 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1316 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1318 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1319 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1405 T68 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1484 T68 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1485 T68 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1503 T68 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2195 T68 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2208 T68 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2213 T68 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2227 T68 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2234 T68 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2238 T68 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2240 T68 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2242 T68 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/
[junit4:junit4]   2> 2242 T68 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1451ef4e
[junit4:junit4]   2> 2282 T68 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/index/
[junit4:junit4]   2> 2283 T68 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2290 T68 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/index forceNew:false
[junit4:junit4]   2> 2298 T68 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@15129dc2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2298 T68 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2300 T68 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 2302 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2303 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 2303 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2304 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 2305 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2305 T68 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2306 T68 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2306 T68 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2307 T68 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2308 T68 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 2308 T68 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 2309 T68 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2309 T68 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2310 T68 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2310 T68 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2311 T68 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2311 T68 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2312 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2312 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2313 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2313 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2314 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2314 T68 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2315 T68 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 2316 T68 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 2321 T68 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2325 T68 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 2327 T68 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMA-MINIMAL.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REPLICATION1.XML, DA_UTF8.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-SPELLCHECKER.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, SOLRCONFIG-FUNCTIONQUERY.XML, STOP-2.TXT, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, SCHEMA-FOLDING.XML, STOPWORDS.TXT, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, BAD-MP-SOLRCONFIG.XML, FRENCHARTICLES.TXT, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, SOLRCONFIG-XINCLUDE.XML, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 2330 T68 oass.SolrIndexSearcher.<init> Opening Searcher@16562d4b main
[junit4:junit4]   2> 2331 T68 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2332 T68 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2332 T68 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 2341 T68 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2386 T68 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2386 T68 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2387 T68 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2387 T68 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2387 T68 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2388 T68 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2388 T68 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2388 T68 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2389 T68 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2389 T68 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2404 T83 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16562d4b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2407 T68 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2408 T68 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2408 T68 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2460 T68 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2478 T68 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2480 T68 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:58229
[junit4:junit4]   2> 2481 T68 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2481 T68 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2481 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2508 T68 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2508 T68 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2509 T68 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2509 T68 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2509 T68 oasc.CoreContainer.<init> New CoreContainer 172802398
[junit4:junit4]   2> 2510 T68 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2510 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2529 T68 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2544 T68 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2545 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2546 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2546 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2594 T68 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2655 T68 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2656 T68 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2666 T68 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 3275 T68 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 3287 T68 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 3291 T68 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 3304 T68 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3310 T68 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3314 T68 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3316 T68 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3317 T68 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/shard0/data/
[junit4:junit4]   2> 3317 T68 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1451ef4e
[junit4:junit4]   2> 3318 T68 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/shard0/data/index/
[junit4:junit4]   2> 3319 T68 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 3320 T68 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/shard0/data/index forceNew:false
[junit4:junit4]   2> 3326 T68 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/shard0/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@7b7a7861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3327 T68 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3328 T68 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 3330 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 3331 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 3331 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 3331 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 3332 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 3332 T68 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 3333 T68 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 3334 T68 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 3334 T68 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 3335 T68 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 3335 T68 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 3336 T68 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 3336 T68 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 3337 T68 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 3337 T68 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 3338 T68 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 3338 T68 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3339 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3340 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3340 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3341 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3343 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3343 T68 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3344 T68 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 3344 T68 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 3350 T68 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 3354 T68 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 3355 T68 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMA-MINIMAL.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REPLICATION1.XML, DA_UTF8.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-SPELLCHECKER.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, SOLRCONFIG-FUNCTIONQUERY.XML, STOP-2.TXT, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, SCHEMA-FOLDING.XML, STOPWORDS.TXT, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, BAD-MP-SOLRCONFIG.XML, FRENCHARTICLES.TXT, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, SOLRCONFIG-XINCLUDE.XML, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 3358 T68 oass.SolrIndexSearcher.<init> Opening Searcher@3bbc8de2 main
[junit4:junit4]   2> 3359 T68 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 3359 T68 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 3360 T68 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 3366 T68 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 3412 T68 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 3412 T68 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3413 T68 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 3413 T68 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3413 T68 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3414 T68 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3414 T68 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3414 T68 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3415 T68 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3415 T68 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 3430 T97 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3bbc8de2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 3434 T68 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 3435 T68 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3435 T68 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3435 T68 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C15 name=collection1 org.apache.solr.core.SolrCore@5889b953
[junit4:junit4]   2> 4501 T82 C15 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 4506 T82 C15 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@15129dc2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4507 T82 C15 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4508 T82 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 45
[junit4:junit4]   2> ASYNC  NEW_CORE C16 name=collection1 org.apache.solr.core.SolrCore@567544a6
[junit4:junit4]   2> 4674 T94 C16 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 4678 T94 C16 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/shard0/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@7b7a7861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4679 T94 C16 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4680 T94 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 6
[junit4:junit4]   2> 4688 T80 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 4
[junit4:junit4]   2> 4694 T96 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 2
[junit4:junit4]   2> 4699 T81 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 1
[junit4:junit4]   2> 4704 T95 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 2
[junit4:junit4]   2> 4709 T82 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 2
[junit4:junit4]   2> 4715 T94 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 2
[junit4:junit4]   2> 4719 T82 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 1
[junit4:junit4]   2> 4724 T96 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 1
[junit4:junit4]   2> 4729 T81 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 2
[junit4:junit4]   2> 4734 T95 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 2
[junit4:junit4]   2> 4739 T80 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 2
[junit4:junit4]   2> 4744 T94 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 1
[junit4:junit4]   2> 4750 T82 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 1
[junit4:junit4]   2> 4756 T96 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 2
[junit4:junit4]   2> 4760 T81 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 1
[junit4:junit4]   2> 4763 T95 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 1
[junit4:junit4]   2> 4768 T82 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 2
[junit4:junit4]   2> 4774 T94 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 2
[junit4:junit4]   2> 4785 T80 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 7
[junit4:junit4]   2> 4796 T96 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 7
[junit4:junit4]   2> 4802 T81 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 2
[junit4:junit4]   2> 4808 T95 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 2
[junit4:junit4]   2> 4813 T82 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[13]} 0 1
[junit4:junit4]   2> 4819 T94 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[13]} 0 2
[junit4:junit4]   2> 4824 T80 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 1
[junit4:junit4]   2> 4829 T96 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 2
[junit4:junit4]   2> 4833 T81 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 2
[junit4:junit4]   2> 4838 T95 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 2
[junit4:junit4]   2> 4843 T82 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 2
[junit4:junit4]   2> 4847 T94 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 1
[junit4:junit4]   2> 4854 T80 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 2
[junit4:junit4]   2> 4859 T96 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 2
[junit4:junit4]   2> 4864 T81 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 2
[junit4:junit4]   2> 4871 T95 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 2
[junit4:junit4]   2> 4877 T82 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 1
[junit4:junit4]   2> 4882 T94 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 2
[junit4:junit4]   2> 4887 T80 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 1
[junit4:junit4]   2> 4892 T96 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 1
[junit4:junit4]   2> 4902 T81 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 8
[junit4:junit4]   2> 4915 T95 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 9
[junit4:junit4]   2> 4923 T82 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 3
[junit4:junit4]   2> 4931 T94 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 2
[junit4:junit4]   2> 4938 T80 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 2
[junit4:junit4]   2> 4944 T96 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 2
[junit4:junit4]   2> 4950 T81 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 2
[junit4:junit4]   2> 4956 T95 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 2
[junit4:junit4]   2> 4961 T82 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 1
[junit4:junit4]   2> 4967 T94 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 1
[junit4:junit4]   2> 5015 T80 C15 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5170 T80 C15 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@15129dc2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@15129dc2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2,filenames=[_2_Lucene41_0.pos, _2.si, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _0_Lucene41_0.pos, _2_Lucene41_0.tip, _1.fnm, _2_nrm.cfe, _2_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1_nrm.cfs, _0.fnm, _1_Lucene41_0.tim, _2.fdt, _1_Lucene41_0.tip, _2_Lucene41_0.doc, _0_nrm.cfs, _2.fdx, _2.fnm, _1_nrm.cfe, _1_Lucene41_0.pos, _1.fdx, _0_Lucene41_0.doc, _1.fdt, _1.si, _0_nrm.cfe, segments_2, _0.fdx, _0.fdt]
[junit4:junit4]   2> 5170 T80 C15 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 5196 T80 C15 oass.SolrIndexSearcher.<init> Opening Searcher@5d325877 main
[junit4:junit4]   2> 5197 T80 C15 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5199 T83 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5d325877 main{StandardDirectoryReader(segments_2:8 _0(4.1):C10 _1(4.1):C10 _2(4.1):C4)}
[junit4:junit4]   2> 5200 T80 C15 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 185
[junit4:junit4]   2> 5204 T96 C16 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5333 T96 C16 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/shard0/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@7b7a7861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/shard0/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@7b7a7861; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2,filenames=[_2_Lucene41_0.pos, _2.si, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _0_Lucene41_0.pos, _2_Lucene41_0.tip, _1.fnm, _2_nrm.cfe, _2_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1_nrm.cfs, _0.fnm, _1_Lucene41_0.tim, _2.fdt, _1_Lucene41_0.tip, _2_Lucene41_0.doc, _0_nrm.cfs, _2.fdx, _2.fnm, _1_nrm.cfe, _1_Lucene41_0.pos, _1.fdx, _0_Lucene41_0.doc, _1.fdt, _1.si, _0_nrm.cfe, segments_2, _0.fdx, _0.fdt]
[junit4:junit4]   2> 5333 T96 C16 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 5365 T96 C16 oass.SolrIndexSearcher.<init> Opening Searcher@29cf542 main
[junit4:junit4]   2> 5366 T96 C16 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5368 T97 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@29cf542 main{StandardDirectoryReader(segments_2:8 _0(4.1):C10 _1(4.1):C10 _2(4.1):C4)}
[junit4:junit4]   2> 5368 T96 C16 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 164
[junit4:junit4]   2> 5558 T81 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&q=*:*&qt=spellCheckCompRH&wt=javabin&spellcheck.build=true&version=2} hits=24 status=0 QTime=154 
[junit4:junit4]   2> 5755 T96 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&spellcheck.extendedResults=false&distrib=false&wt=javabin&rows=10&spellcheck.onlyMorePopular=false&version=2&fl=id,score&shard.url=127.0.0.1:58229/solr&NOW=1352187413863&start=0&q=*:*&spellcheck.count=5&qt=spellCheckCompRH&isShard=true&fsv=true&spellcheck.build=true} hits=24 status=0 QTime=88 
[junit4:junit4]   2> 5793 T96 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.extendedResults=false&shards.qt=spellCheckCompRH&ids=3,2,1,10,7,6,5,4,9,8&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&version=2&shard.url=127.0.0.1:58229/solr&NOW=1352187413863&q=*:*&spellcheck.count=1&qt=spellCheckCompRH&isShard=true&spellcheck.build=true} status=0 QTime=4 
[junit4:junit4]   2> 5803 T94 C16 oashc.SpellCheckComponent.finishStage 127.0.0.1:58229/solr null
[junit4:junit4]   2> 5804 T94 C16 oashc.SpellCheckComponent.finishStage 127.0.0.1:58229/solr null
[junit4:junit4]   2> 5804 T94 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards=127.0.0.1:58229/solr&shards.qt=spellCheckCompRH&q=*:*&qt=spellCheckCompRH&wt=javabin&spellcheck.build=true&version=2} hits=24 status=0 QTime=239 
[junit4:junit4]   2> 5920 T80 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards.qt=spellCheckCompRH_Direct&q=*:*&spellcheck.q=toyata&distrib=false&qt=spellCheckCompRH_Direct&wt=javabin&version=2} hits=24 status=0 QTime=104 
[junit4:junit4]   2> 5961 T94 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=false&shards.qt=spellCheckCompRH_Direct&spellcheck.q=toyata&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=127.0.0.1:58229/solr&NOW=1352187414230&start=0&q=*:*&spellcheck.dictionary=direct&spellcheck.count=5&qt=spellCheckCompRH_Direct&isShard=true&fsv=true} hits=24 status=0 QTime=25 
[junit4:junit4]   2> 5969 T95 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.extendedResults=false&shards.qt=spellCheckCompRH_Direct&ids=3,2,1,10,7,6,5,4,9,8&spellcheck.q=toyata&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&version=2&fl=id,lowerfilt&shard.url=127.0.0.1:58229/solr&NOW=1352187414230&q=*:*&spellcheck.dictionary=direct&spellcheck.count=1&qt=spellCheckCompRH_Direct&isShard=true} status=0 QTime=3 
[junit4:junit4]   2> 5972 T96 C16 oashc.SpellCheckComponent.finishStage 127.0.0.1:58229/solr {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,suggestion=[toyota]}}}
[junit4:junit4]   2> 5973 T96 C16 oashc.SpellCheckComponent.finishStage 127.0.0.1:58229/solr null
[junit4:junit4]   2> 5973 T96 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards=127.0.0.1:58229/solr&shards.qt=spellCheckCompRH_Direct&q=*:*&spellcheck.q=toyata&qt=spellCheckCompRH_Direct&wt=javabin&version=2} hits=24 status=0 QTime=41 
[junit4:junit4]   2> 5994 T81 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards.qt=spellCheckCompRH_Direct&spellcheck.extendedResults=true&q=*:*&spellcheck.q=toyata&distrib=false&qt=spellCheckCompRH_Direct&wt=javabin&version=2} hits=24 status=0 QTime=17 
[junit4:junit4]   2> 6034 T95 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH_Direct&spellcheck.q=toyata&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=127.0.0.1:58229/solr&NOW=1352187414297&start=0&q=*:*&spellcheck.dictionary=direct&spellcheck.count=5&qt=spellCheckCompRH_Direct&isShard=true&fsv=true} hits=24 status=0 QTime=30 
[junit4:junit4]   2> 6041 T96 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH_Direct&ids=3,2,1,10,7,6,5,4,9,8&spellcheck.q=toyata&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&version=2&fl=id,lowerfilt&shard.url=127.0.0.1:58229/solr&NOW=1352187414297&q=*:*&spellcheck.dictionary=direct&spellcheck.count=1&qt=spellCheckCompRH_Direct&isShard=true} status=0 QTime=2 
[junit4:junit4]   2> 6042 T94 C16 oashc.SpellCheckComponent.finishStage 127.0.0.1:58229/solr {suggestions={toyata={numFound=1,startOffset=0,endOffset=6,origFreq=0,suggestion=[{word=toyota,freq=1}]},correctlySpelled=true}}
[junit4:junit4]   2> 6043 T94 C16 oashc.SpellCheckComponent.finishStage 127.0.0.1:58229/solr null
[junit4:junit4]   2> 6044 T94 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards=127.0.0.1:58229/solr&shards.qt=spellCheckCompRH_Direct&spellcheck.extendedResults=true&q=*:*&spellcheck.q=toyata&qt=spellCheckCompRH_Direct&wt=javabin&version=2} hits=24 status=0 QTime=45 
[junit4:junit4]   2> 6066 T82 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards.qt=spellCheckCompRH_Direct&spellcheck.extendedResults=true&q=*:*&spellcheck.q=bluo&distrib=false&spellcheck.count=4&qt=spellCheckCompRH_Direct&wt=javabin&version=2} hits=24 status=0 QTime=17 
[junit4:junit4]   2> 6263 T96 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH_Direct&spellcheck.q=bluo&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=[ff01::114]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:58229/solr|[ff01::083]:33332/solr&NOW=1352187414371&start=0&q=*:*&spellcheck.dictionary=direct&spellcheck.count=5&qt=spellCheckCompRH_Direct&isShard=true&fsv=true} hits=24 status=0 QTime=7 
[junit4:junit4]   2> 6273 T94 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH_Direct&ids=3,2,1,10,7,6,5,4,9,8&spellcheck.q=bluo&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&version=2&fl=id,lowerfilt&shard.url=[ff01::114]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:58229/solr|[ff01::083]:33332/solr&NOW=1352187414371&q=*:*&spellcheck.dictionary=direct&spellcheck.count=4&qt=spellCheckCompRH_Direct&isShard=true} status=0 QTime=4 
[junit4:junit4]   2> 6275 T95 C16 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:58229/solr|[ff01::083]:33332/solr {suggestions={bluo={numFound=1,startOffset=0,endOffset=4,origFreq=0,suggestion=[{word=blue,freq=1}]},correctlySpelled=true}}
[junit4:junit4]   2> 6275 T95 C16 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:58229/solr|[ff01::083]:33332/solr null
[junit4:junit4]   2> 6275 T95 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards=[ff01::114]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:58229/solr|[ff01::083]:33332/solr&shards.qt=spellCheckCompRH_Direct&spellcheck.extendedResults=true&q=*:*&spellcheck.q=bluo&spellcheck.count=4&qt=spellCheckCompRH_Direct&wt=javabin&version=2} hits=24 status=0 QTime=202 
[junit4:junit4]   2> 6325 T80 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards.qt=spellCheckCompRH_Direct&spellcheck.extendedResults=true&q=The+quick+reb+fox+jumped+over+the+lazy+brown+dogs&distrib=false&spellcheck.count=4&qt=spellCheckCompRH_Direct&spellcheck.collate=true&wt=javabin&version=2} hits=0 status=0 QTime=45 
[junit4:junit4]   2> 6360 T95 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH_Direct&distrib=false&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=[ff01::114]:33332/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:58229/solr&NOW=1352187414632&start=0&q=The+quick+reb+fox+jumped+over+the+lazy+brown+dogs&spellcheck.dictionary=direct&spellcheck.count=5&qt=spellCheckCompRH_Direct&isShard=true&fsv=true} hits=0 status=0 QTime=21 
[junit4:junit4]   2> 6362 T94 C16 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:58229/solr {suggestions={reb={numFound=1,startOffset=10,endOffset=13,origFreq=0,suggestion=[{word=red,freq=13}]},dogs={numFound=1,startOffset=45,endOffset=49,origFreq=0,suggestion=[{word=dogs.,freq=13}]},correctlySpelled=false,collation=The quick red fox jumped over the lazy brown dogs.}}
[junit4:junit4]   2> 6363 T94 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards=[ff01::114]:33332/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:58229/solr&shards.qt=spellCheckCompRH_Direct&spellcheck.extendedResults=true&q=The+quick+reb+fox+jumped+over+the+lazy+brown+dogs&spellcheck.count=4&qt=spellCheckCompRH_Direct&spellcheck.collate=true&wt=javabin&version=2} hits=0 status=0 QTime=29 
[junit4:junit4]   2> 6384 T82 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.collateExtendedResults=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH_Direct&spellcheck.maxCollations=10&spellcheck.maxCollationTries=10&distrib=false&wt=javabin&spellcheck.collate=true&version=2&fl=id,lowerfilt&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH_Direct} hits=0 status=0 QTime=17 
[junit4:junit4]   2> 6425 T95 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH_Direct&spellcheck.maxCollationTries=10&distrib=false&version=2&fl=id,score&shard.url=127.0.0.1:58229/solr&NOW=1352187414696&spellcheck.count=10&qt=spellCheckCompRH_Direct&fsv=true&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollations=10&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&start=0&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.dictionary=direct&isShard=true} hits=0 status=0 QTime=21 
[junit4:junit4]   2> 6427 T96 C16 oashc.SpellCheckComponent.finishStage 127.0.0.1:58229/solr {suggestions={quock={numFound=2,startOffset=12,endOffset=17,origFreq=0,suggestion=[{word=quick,freq=1}, {word=quote,freq=12}]},reb={numFound=1,startOffset=19,endOffset=22,origFreq=0,suggestion=[{word=red,freq=13}]},correctlySpelled=false,collation={collationQuery=lowerfilt:(+quick +red),hits=1,misspellingsAndCorrections={quock=quick,reb=red},collationInternalRank=0},collation={collationQuery=lowerfilt:(+quote +red),hits=12,misspellingsAndCorrections={quock=quote,reb=red},collationInternalRank=1}}}
[junit4:junit4]   2> 6428 T96 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.collateExtendedResults=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH_Direct&spellcheck.maxCollations=10&spellcheck.maxCollationTries=10&wt=javabin&spellcheck.collate=true&version=2&shards=127.0.0.1:58229/solr&fl=id,lowerfilt&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH_Direct} hits=0 status=0 QTime=30 
[junit4:junit4]   2> 6440 T81 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH_Direct&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=false&spellcheck.maxCollationTries=10&spellcheck.maxCollations=10&distrib=false&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH_Direct} hits=0 status=0 QTime=8 
[junit4:junit4]   2> 6465 T95 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH_Direct&spellcheck.maxCollationTries=10&distrib=false&version=2&fl=id,score&shard.url=[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:58229/solr&NOW=1352187414743&spellcheck.count=10&qt=spellCheckCompRH_Direct&fsv=true&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=false&spellcheck.maxCollations=10&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&start=0&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.dictionary=direct&isShard=true} hits=0 status=0 QTime=16 
[junit4:junit4]   2> 6467 T94 C16 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:58229/solr {suggestions={quock={numFound=2,startOffset=12,endOffset=17,origFreq=0,suggestion=[{word=quick,freq=1}, {word=quote,freq=12}]},reb={numFound=1,startOffset=19,endOffset=22,origFreq=0,suggestion=[{word=red,freq=13}]},correctlySpelled=false,collation=lowerfilt:(+quick +red),collationInternalRank=0,collation=lowerfilt:(+quote +red),collationInternalRank=1}}
[junit4:junit4]   2> 6468 T94 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH_Direct&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=false&spellcheck.maxCollationTries=10&spellcheck.maxCollations=10&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&shards=[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:58229/solr&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH_Direct} hits=0 status=0 QTime=23 
[junit4:junit4]   2> 6486 T80 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH_Direct&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=false&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH_Direct} hits=0 status=0 QTime=14 
[junit4:junit4]   2> 6518 T96 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH_Direct&spellcheck.maxCollationTries=0&distrib=false&version=2&fl=id,score&shard.url=[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:58229/solr&NOW=1352187414789&spellcheck.count=10&qt=spellCheckCompRH_Direct&fsv=true&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=false&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&start=0&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.dictionary=direct&isShard=true} hits=0 status=0 QTime=21 
[junit4:junit4]   2> 6519 T95 C16 oashc.SpellCheckComponent.finishStage [ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:58229/solr {suggestions={quock={numFound=2,startOffset=12,endOffset=17,origFreq=0,suggestion=[{word=quick,freq=1}, {word=quote,freq=12}]},reb={numFound=1,startOffset=19,endOffset=22,origFreq=0,suggestion=[{word=red,freq=13}]},correctlySpelled=false,collation=lowerfilt:(+quick +red)}}
[junit4:junit4]   2> 6520 T95 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH_Direct&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=false&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&shards=[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:58229/solr&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH_Direct} hits=0 status=0 QTime=29 
[junit4:junit4]   2> 6528 T82 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.maxResultsForSuggest=10&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&q=lowerfilt:("quote+red+fox")&spellcheck.count=10&qt=spellCheckCompRH&spellcheck.alternativeTermCount=5} hits=12 status=0 QTime=3 
[junit4:junit4]   2> 6544 T96 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&spellcheck.maxCollationTries=10&distrib=false&version=2&fl=id,score&shard.url=[ff01::114]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:58229/solr|[ff01::083]:33332/solr&NOW=1352187414832&spellcheck.count=10&qt=spellCheckCompRH&fsv=true&spellcheck.alternativeTermCount=5&spellcheck.maxResultsForSuggest=10&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&start=0&q=lowerfilt:("quote+red+fox")&isShard=true} hits=12 status=0 QTime=3 
[junit4:junit4]   2> 6555 T95 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.maxResultsForSuggest=10&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&ids=21,20,19,17,22,18,15,16,13,14&distrib=false&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&version=2&fl=id,lowerfilt&shard.url=[ff01::114]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:58229/solr|[ff01::083]:33332/solr&NOW=1352187414832&q=lowerfilt:("quote+red+fox")&spellcheck.count=10&qt=spellCheckCompRH&isShard=true&spellcheck.alternativeTermCount=5} status=0 QTime=4 
[junit4:junit4]   2> 6559 T94 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.maxResultsForSuggest=10&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&shards=[ff01::114]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:58229/solr|[ff01::083]:33332/solr&q=lowerfilt:("quote+red+fox")&spellcheck.count=10&qt=spellCheckCompRH&spellcheck.alternativeTermCount=5} hits=12 status=0 QTime=24 
[junit4:junit4]   2> 6578 T81 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.maxResultsForSuggest=10&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&q=lowerfilt:("rod+fix")&spellcheck.count=10&qt=spellCheckCompRH&spellcheck.alternativeTermCount=5} hits=1 status=0 QTime=14 
[junit4:junit4]   2> 6600 T95 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&spellcheck.maxCollationTries=10&distrib=false&version=2&fl=id,score&shard.url=127.0.0.1:58229/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr&NOW=1352187414881&spellcheck.count=10&qt=spellCheckCompRH&fsv=true&spellcheck.alternativeTermCount=5&spellcheck.maxResultsForSuggest=10&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&start=0&q=lowerfilt:("rod+fix")&isShard=true} hits=1 status=0 QTime=12 
[junit4:junit4]   2> 6606 T94 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.maxResultsForSuggest=10&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&ids=25&distrib=false&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&version=2&fl=id,lowerfilt&shard.url=127.0.0.1:58229/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr&NOW=1352187414881&q=lowerfilt:("rod+fix")&spellcheck.count=10&qt=spellCheckCompRH&isShard=true&spellcheck.alternativeTermCount=5} status=0 QTime=1 
[junit4:junit4]   2> 6607 T96 C16 oashc.SpellCheckComponent.finishStage 127.0.0.1:58229/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr {suggestions={rod={numFound=2,startOffset=12,endOffset=15,origFreq=1,suggestion=[{word=red,freq=13}, {word=ford,freq=1}]},fix={numFound=1,startOffset=16,endOffset=19,origFreq=1,suggestion=[{word=fox,freq=13}]},correctlySpelled=false,collation={collationQuery=lowerfilt:("red fox"),hits=13,misspellingsAndCorrections={rod=red,fix=fox},collationInternalRank=2}}}
[junit4:junit4]   2> 6608 T96 C16 oashc.SpellCheckComponent.finishStage 127.0.0.1:58229/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr null
[junit4:junit4]   2> 6608 T96 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.maxResultsForSuggest=10&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&shards=127.0.0.1:58229/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr&q=lowerfilt:("rod+fix")&spellcheck.count=10&qt=spellCheckCompRH&spellcheck.alternativeTermCount=5} hits=1 status=0 QTime=25 
[junit4:junit4]   2> 6685 T80 C15 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckWithWordbreak_Direct&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&q=lowerfilt:(%2Bquock+%2Bredfox+%2Bjum+%2Bped)&spellcheck.count=10&qt=spellCheckWithWordbreak_Direct} hits=0 status=0 QTime=73 
[junit4:junit4]   2> 6707 T96 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckWithWordbreak_Direct&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&rows=10&version=2&fl=id,score&shard.url=[ff01::083]:33332/solr|127.0.0.1:58229/solr|[ff01::213]:33332/solr|[ff01::114]:33332/solr&NOW=1352187414988&start=0&q=lowerfilt:(%2Bquock+%2Bredfox+%2Bjum+%2Bped)&spellcheck.dictionary=direct&spellcheck.dictionary=wordbreak&spellcheck.count=10&qt=spellCheckWithWordbreak_Direct&isShard=true&fsv=true} hits=0 status=0 QTime=13 
[junit4:junit4]   2> 6709 T95 C16 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/solr|127.0.0.1:58229/solr|[ff01::213]:33332/solr|[ff01::114]:33332/solr {suggestions={quock={numFound=2,startOffset=12,endOffset=17,origFreq=0,suggestion=[{word=quick,freq=1}, {word=quote,freq=12}]},redfox={numFound=1,startOffset=19,endOffset=25,origFreq=0,suggestion=[{word=red fox,freq=13}]},jum ped={numFound=1,startOffset=27,endOffset=35,origFreq=0,suggestion=[{word=jumped,freq=13}]},correctlySpelled=false,collation={collationQuery=lowerfilt:(+quick (+red +fox) +jumped),hits=0,misspellingsAndCorrections={quock=quick,redfox=red fox,jum ped=jumped}}}}
[junit4:junit4]   2> 6711 T95 C16 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckWithWordbreak_Direct&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&shards=[ff01::083]:33332/solr|127.0.0.1:58229/solr|[ff01::213]:33332/solr|[ff01::114]:33332/solr&q=lowerfilt:(%2Bquock+%2Bredfox+%2Bjum+%2Bped)&spellcheck.count=10&qt=spellCheckWithWordbreak_Direct} hits=0 status=0 QTime=21 
[junit4:junit4]   2> 6726 T68 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=703213836
[junit4:junit4]   2> 6728 T68 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5889b953
[junit4:junit4]   2> 6735 T68 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=24,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 6736 T68 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 6737 T68 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 6737 T68 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 6739 T68 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 6741 T68 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 6842 T68 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 6887 T68 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=172802398
[junit4:junit4]   2> 6888 T68 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@567544a6
[junit4:junit4]   2> 6897 T68 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=24,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 6898 T68 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 6898 T68 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 6899 T68 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 6905 T68 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 6910 T68 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 7011 T68 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 7013 T68 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 7015 T68 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:63574
[junit4:junit4]   2> 7016 T68 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7016 T68 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 7017 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7049 T68 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 7050 T68 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7050 T68 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 7051 T68 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 7051 T68 oasc.CoreContainer.<init> New CoreContainer 246360237
[junit4:junit4]   2> 7052 T68 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7052 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7074 T68 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 7090 T68 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 7091 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 7092 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 7093 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 7150 T68 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 7224 T68 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 7225 T68 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 7236 T68 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 7785 T68 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 7798 T68 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 7803 T68 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 7816 T68 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7822 T68 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7827 T68 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7829 T68 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7830 T68 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/
[junit4:junit4]   2> 7831 T68 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1451ef4e
[junit4:junit4]   2> 7832 T68 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/index/
[junit4:junit4]   2> 7833 T68 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 7836 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 7837 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 7837 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 7838 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 7839 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 7839 T68 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 7840 T68 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 7841 T68 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 7842 T68 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 7842 T68 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 7843 T68 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 7844 T68 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 7844 T68 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 7845 T68 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 7846 T68 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 7846 T68 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 7847 T68 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7848 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7849 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7849 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7850 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7851 T68 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7851 T68 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7852 T68 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 7853 T68 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 7861 T68 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 7863 T68 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 7865 T68 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMA-MINIMAL.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REPLICATION1.XML, DA_UTF8.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-SPELLCHECKER.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, SOLRCONFIG-FUNCTIONQUERY.XML, STOP-2.TXT, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, SCHEMA-FOLDING.XML, STOPWORDS.TXT, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, BAD-MP-SOLRCONFIG.XML, FRENCHARTICLES.TXT, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, SOLRCONFIG-XINCLUDE.XML, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 7869 T68 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/control/data/index forceNew:false
[junit4:junit4]   2> 7889 T68 oass.SolrIndexSearcher.<init> Opening Searcher@45263910 main
[junit4:junit4]   2> 7890 T68 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 7891 T68 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7891 T68 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 7900 T68 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 7913 T68 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 7914 T68 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 7914 T68 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 7914 T68 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 7915 T68 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 7915 T68 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 7915 T68 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 7916 T68 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 7916 T68 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 7917 T68 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 7931 T113 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@45263910 main{StandardDirectoryReader(segments_2:8 _0(4.1):C10 _1(4.1):C10 _2(4.1):C4)}
[junit4:junit4]   2> 7934 T68 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 7935 T68 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 7935 T68 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 7935 T68 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 7945 T68 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 7947 T68 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:50874
[junit4:junit4]   2> 7948 T68 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7948 T68 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 7948 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7973 T68 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 7973 T68 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7974 T68 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 7974 T68 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 7974 T68 oasc.CoreContainer.<init> New CoreContainer 1636237639
[junit4:junit4]   2> 7975 T68 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7975 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7993 T68 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8009 T68 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 8010 T68 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 8011 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 8012 T68 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 8051 T68 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 8103 T68 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 8104 T68 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 8113 T68 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 8562 T68 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 8571 T68 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 8575 T68 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 8587 T68 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 8593 T68 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 8597 T68 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 8599 T68 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 8601 T68 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/shard0/data/
[junit4:junit4]   2> 8601 T68 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1451ef4e
[junit4:junit4]   2> 8602 T68 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1352187408305/shard0/data/index/
[junit4:junit4]   2> 8603 T68 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 8606 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 8606 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 8607 T68 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"

[...truncated too long message...]

t4]   2> 	        at sun.nio.ch.KqueueArrayWrapper.poll(KqueueArrayWrapper.java:146)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueSelectorImpl.doSelect(KqueueSelectorImpl.java:69)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
[junit4:junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:531)
[junit4:junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  21) Thread[id=364, name=TEST-FullSolrCloudDistribCmdsTest.testDistribSearch-seed#[38DF40773F94B8B2]-SendThread(localhost:26547), state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueArrayWrapper.kevent(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueArrayWrapper.poll(KqueueArrayWrapper.java:146)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueSelectorImpl.doSelect(KqueueSelectorImpl.java:69)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4:junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1131)
[junit4:junit4]   2> 	  22) Thread[id=326, name=qtp446179646-326 Acceptor2 SelectChannelConnector@0.0.0.0:26556, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  23) Thread[id=292, name=qtp792820521-292 Acceptor2 SelectChannelConnector@0.0.0.0:26551, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  24) Thread[id=356, name=qtp969284083-356 Acceptor0 SelectChannelConnector@0.0.0.0:26563, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  25) Thread[id=291, name=qtp792820521-291 Acceptor1 SelectChannelConnector@0.0.0.0:26551, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  26) Thread[id=369, name=TEST-FullSolrCloudDistribCmdsTest.testDistribSearch-seed#[38DF40773F94B8B2]-SendThread(localhost:26547), state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueArrayWrapper.kevent(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueArrayWrapper.poll(KqueueArrayWrapper.java:146)
[junit4:junit4]   2> 	        at sun.nio.ch.KqueueSelectorImpl.doSelect(KqueueSelectorImpl.java:69)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4:junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1131)
[junit4:junit4]   2> 	  27) Thread[id=357, name=qtp969284083-357 Acceptor1 SelectChannelConnector@0.0.0.0:26563, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  28) Thread[id=323, name=qtp446179646-323 Selector3, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:82)
[junit4:junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4:junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:564)
[junit4:junit4]   2> 	        at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:285)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene3x, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=tr, timezone=America/Moncton
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=37,free=241344280,total=427163648
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestAnalyzedSuggestions, DisMaxRequestHandlerTest, UUIDFieldTest, TestFastOutputStream, FieldMutatingUpdateProcessorTest, TestStressVersions, StatelessScriptUpdateProcessorFactoryTest, QueryEqualityTest, DebugComponentTest, MultiTermTest, TestStressRecovery, TestSurroundQueryParser, LeaderElectionTest, TestSolrDeletionPolicy2, FullSolrCloudDistribCmdsTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.seed=38DF40773F94B8B2 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=tr -Dtests.timezone=America/Moncton -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR   0.00s J1 | FullSolrCloudDistribCmdsTest (suite) <<<
[junit4:junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([38DF40773F94B8B2]:0)
[junit4:junit4] Completed on J1 in 7224.23s, 1 test, 2 errors <<< FAILURES!

[...truncated 144 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 3 suite-level errors, 1 error, 708 ignored

Total time: 137 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 835 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/835/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
64 threads leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandler:     1) Thread[id=861, name=qtp431075824-861 Acceptor1 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=718, name=qtp1025719152-718 Acceptor3 SelectChannelConnector@0.0.0.0:47379, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=707, name=qtp1275406235-707 Acceptor2 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=686, name=qtp1151506535-686 Acceptor1 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=772, name=qtp1587559111-772 Acceptor1 SelectChannelConnector@0.0.0.0:29265, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=773, name=qtp1587559111-773 Acceptor2 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=840, name=qtp240734214-840 Acceptor0 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=863, name=qtp431075824-863 Acceptor3 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=816, name=qtp95968411-816 Acceptor2 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=715, name=qtp1025719152-715 Acceptor0 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=661, name=qtp983787684-661 Acceptor2 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=659, name=qtp983787684-659 Acceptor0 SelectChannelConnector@0.0.0.0:10177, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=843, name=qtp240734214-843 Acceptor3 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=628, name=qtp1729456457-628 Acceptor0 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=674, name=qtp804686489-674 Acceptor1 SelectChannelConnector@0.0.0.0:32394, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=783, name=qtp1152181387-783 Acceptor2 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=616, name=qtp1296726923-616 Acceptor3 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=629, name=qtp1729456457-629 Acceptor1 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=815, name=qtp95968411-815 Acceptor1 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=603, name=qtp520031133-603 Acceptor0 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=803, name=qtp1294089382-803 Acceptor3 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=687, name=qtp1151506535-687 Acceptor2 SelectChannelConnector@0.0.0.0:49557, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=641, name=qtp497288441-641 Acceptor3 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=760, name=qtp1088216030-760 Acceptor0 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=802, name=qtp1294089382-802 Acceptor2 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=675, name=qtp804686489-675 Acceptor2 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=662, name=qtp983787684-662 Acceptor3 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=660, name=qtp983787684-660 Acceptor1 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=784, name=qtp1152181387-784 Acceptor3 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=762, name=qtp1088216030-762 Acceptor2 SelectChannelConnector@0.0.0.0:16754, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=673, name=qtp804686489-673 Acceptor0 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=614, name=qtp1296726923-614 Acceptor1 SelectChannelConnector@0.0.0.0:34414, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=782, name=qtp1152181387-782 Acceptor1 SelectChannelConnector@0.0.0.0:25162, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=717, name=qtp1025719152-717 Acceptor2 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=708, name=qtp1275406235-708 Acceptor3 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=814, name=qtp95968411-814 Acceptor0 SelectChannelConnector@0.0.0.0:64705, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   37) Thread[id=705, name=qtp1275406235-705 Acceptor0 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   38) Thread[id=688, name=qtp1151506535-688 Acceptor3 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   39) Thread[id=706, name=qtp1275406235-706 Acceptor1 SelectChannelConnector@0.0.0.0:51732, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   40) Thread[id=860, name=qtp431075824-860 Acceptor0 SelectChannelConnector@0.0.0.0:31942, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   41) Thread[id=841, name=qtp240734214-841 Acceptor1 SelectChannelConnector@0.0.0.0:13986, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   42) Thread[id=763, name=qtp1088216030-763 Acceptor3 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   43) Thread[id=605, name=qtp520031133-605 Acceptor2 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   44) Thread[id=630, name=qtp1729456457-630 Acceptor2 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   45) Thread[id=615, name=qtp1296726923-615 Acceptor2 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   46) Thread[id=606, name=qtp520031133-606 Acceptor3 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   47) Thread[id=781, name=qtp1152181387-781 Acceptor0 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   48) Thread[id=639, name=qtp497288441-639 Acceptor1 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   49) Thread[id=842, name=qtp240734214-842 Acceptor2 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   50) Thread[id=613, name=qtp1296726923-613 Acceptor0 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   51) Thread[id=800, name=qtp1294089382-800 Acceptor0 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   52) Thread[id=676, name=qtp804686489-676 Acceptor3 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   53) Thread[id=631, name=qtp1729456457-631 Acceptor3 SelectChannelConnector@0.0.0.0:21124, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   54) Thread[id=771, name=qtp1587559111-771 Acceptor0 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   55) Thread[id=774, name=qtp1587559111-774 Acceptor3 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   56) Thread[id=817, name=qtp95968411-817 Acceptor3 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   57) Thread[id=685, name=qtp1151506535-685 Acceptor0 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   58) Thread[id=761, name=qtp1088216030-761 Acceptor1 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   59) Thread[id=862, name=qtp431075824-862 Acceptor2 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   60) Thread[id=638, name=qtp497288441-638 Acceptor0 SelectChannelConnector@0.0.0.0:36730, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   61) Thread[id=801, name=qtp1294089382-801 Acceptor1 SelectChannelConnector@0.0.0.0:42775, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   62) Thread[id=604, name=qtp520031133-604 Acceptor1 SelectChannelConnector@0.0.0.0:63844, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   63) Thread[id=640, name=qtp497288441-640 Acceptor2 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   64) Thread[id=716, name=qtp1025719152-716 Acceptor1 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 64 threads leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandler: 
   1) Thread[id=861, name=qtp431075824-861 Acceptor1 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=718, name=qtp1025719152-718 Acceptor3 SelectChannelConnector@0.0.0.0:47379, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=707, name=qtp1275406235-707 Acceptor2 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=686, name=qtp1151506535-686 Acceptor1 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=772, name=qtp1587559111-772 Acceptor1 SelectChannelConnector@0.0.0.0:29265, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=773, name=qtp1587559111-773 Acceptor2 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=840, name=qtp240734214-840 Acceptor0 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=863, name=qtp431075824-863 Acceptor3 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=816, name=qtp95968411-816 Acceptor2 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=715, name=qtp1025719152-715 Acceptor0 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=661, name=qtp983787684-661 Acceptor2 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=659, name=qtp983787684-659 Acceptor0 SelectChannelConnector@0.0.0.0:10177, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=843, name=qtp240734214-843 Acceptor3 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=628, name=qtp1729456457-628 Acceptor0 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=674, name=qtp804686489-674 Acceptor1 SelectChannelConnector@0.0.0.0:32394, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=783, name=qtp1152181387-783 Acceptor2 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=616, name=qtp1296726923-616 Acceptor3 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=629, name=qtp1729456457-629 Acceptor1 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=815, name=qtp95968411-815 Acceptor1 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=603, name=qtp520031133-603 Acceptor0 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=803, name=qtp1294089382-803 Acceptor3 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=687, name=qtp1151506535-687 Acceptor2 SelectChannelConnector@0.0.0.0:49557, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=641, name=qtp497288441-641 Acceptor3 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=760, name=qtp1088216030-760 Acceptor0 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=802, name=qtp1294089382-802 Acceptor2 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=675, name=qtp804686489-675 Acceptor2 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=662, name=qtp983787684-662 Acceptor3 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=660, name=qtp983787684-660 Acceptor1 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=784, name=qtp1152181387-784 Acceptor3 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=762, name=qtp1088216030-762 Acceptor2 SelectChannelConnector@0.0.0.0:16754, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=673, name=qtp804686489-673 Acceptor0 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=614, name=qtp1296726923-614 Acceptor1 SelectChannelConnector@0.0.0.0:34414, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=782, name=qtp1152181387-782 Acceptor1 SelectChannelConnector@0.0.0.0:25162, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=717, name=qtp1025719152-717 Acceptor2 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=708, name=qtp1275406235-708 Acceptor3 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=814, name=qtp95968411-814 Acceptor0 SelectChannelConnector@0.0.0.0:64705, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  37) Thread[id=705, name=qtp1275406235-705 Acceptor0 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  38) Thread[id=688, name=qtp1151506535-688 Acceptor3 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  39) Thread[id=706, name=qtp1275406235-706 Acceptor1 SelectChannelConnector@0.0.0.0:51732, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  40) Thread[id=860, name=qtp431075824-860 Acceptor0 SelectChannelConnector@0.0.0.0:31942, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  41) Thread[id=841, name=qtp240734214-841 Acceptor1 SelectChannelConnector@0.0.0.0:13986, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  42) Thread[id=763, name=qtp1088216030-763 Acceptor3 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  43) Thread[id=605, name=qtp520031133-605 Acceptor2 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  44) Thread[id=630, name=qtp1729456457-630 Acceptor2 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  45) Thread[id=615, name=qtp1296726923-615 Acceptor2 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  46) Thread[id=606, name=qtp520031133-606 Acceptor3 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  47) Thread[id=781, name=qtp1152181387-781 Acceptor0 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  48) Thread[id=639, name=qtp497288441-639 Acceptor1 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  49) Thread[id=842, name=qtp240734214-842 Acceptor2 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  50) Thread[id=613, name=qtp1296726923-613 Acceptor0 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  51) Thread[id=800, name=qtp1294089382-800 Acceptor0 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  52) Thread[id=676, name=qtp804686489-676 Acceptor3 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  53) Thread[id=631, name=qtp1729456457-631 Acceptor3 SelectChannelConnector@0.0.0.0:21124, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  54) Thread[id=771, name=qtp1587559111-771 Acceptor0 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  55) Thread[id=774, name=qtp1587559111-774 Acceptor3 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  56) Thread[id=817, name=qtp95968411-817 Acceptor3 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  57) Thread[id=685, name=qtp1151506535-685 Acceptor0 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  58) Thread[id=761, name=qtp1088216030-761 Acceptor1 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  59) Thread[id=862, name=qtp431075824-862 Acceptor2 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  60) Thread[id=638, name=qtp497288441-638 Acceptor0 SelectChannelConnector@0.0.0.0:36730, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  61) Thread[id=801, name=qtp1294089382-801 Acceptor1 SelectChannelConnector@0.0.0.0:42775, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  62) Thread[id=604, name=qtp520031133-604 Acceptor1 SelectChannelConnector@0.0.0.0:63844, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  63) Thread[id=640, name=qtp497288441-640 Acceptor2 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  64) Thread[id=716, name=qtp1025719152-716 Acceptor1 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([88D599395718F613]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=861, name=qtp431075824-861 Acceptor1 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=707, name=qtp1275406235-707 Acceptor2 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=718, name=qtp1025719152-718 Acceptor3 SelectChannelConnector@0.0.0.0:47379, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=686, name=qtp1151506535-686 Acceptor1 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=772, name=qtp1587559111-772 Acceptor1 SelectChannelConnector@0.0.0.0:29265, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=773, name=qtp1587559111-773 Acceptor2 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=840, name=qtp240734214-840 Acceptor0 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=863, name=qtp431075824-863 Acceptor3 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=659, name=qtp983787684-659 Acceptor0 SelectChannelConnector@0.0.0.0:10177, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=661, name=qtp983787684-661 Acceptor2 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=715, name=qtp1025719152-715 Acceptor0 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=816, name=qtp95968411-816 Acceptor2 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=843, name=qtp240734214-843 Acceptor3 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=628, name=qtp1729456457-628 Acceptor0 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=674, name=qtp804686489-674 Acceptor1 SelectChannelConnector@0.0.0.0:32394, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=783, name=qtp1152181387-783 Acceptor2 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=616, name=qtp1296726923-616 Acceptor3 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=629, name=qtp1729456457-629 Acceptor1 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=815, name=qtp95968411-815 Acceptor1 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=603, name=qtp520031133-603 Acceptor0 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=803, name=qtp1294089382-803 Acceptor3 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=687, name=qtp1151506535-687 Acceptor2 SelectChannelConnector@0.0.0.0:49557, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=641, name=qtp497288441-641 Acceptor3 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=760, name=qtp1088216030-760 Acceptor0 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=802, name=qtp1294089382-802 Acceptor2 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=675, name=qtp804686489-675 Acceptor2 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=660, name=qtp983787684-660 Acceptor1 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=662, name=qtp983787684-662 Acceptor3 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=784, name=qtp1152181387-784 Acceptor3 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=762, name=qtp1088216030-762 Acceptor2 SelectChannelConnector@0.0.0.0:16754, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=673, name=qtp804686489-673 Acceptor0 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=614, name=qtp1296726923-614 Acceptor1 SelectChannelConnector@0.0.0.0:34414, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=708, name=qtp1275406235-708 Acceptor3 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=717, name=qtp1025719152-717 Acceptor2 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=782, name=qtp1152181387-782 Acceptor1 SelectChannelConnector@0.0.0.0:25162, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=814, name=qtp95968411-814 Acceptor0 SelectChannelConnector@0.0.0.0:64705, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   37) Thread[id=705, name=qtp1275406235-705 Acceptor0 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   38) Thread[id=688, name=qtp1151506535-688 Acceptor3 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   39) Thread[id=706, name=qtp1275406235-706 Acceptor1 SelectChannelConnector@0.0.0.0:51732, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   40) Thread[id=860, name=qtp431075824-860 Acceptor0 SelectChannelConnector@0.0.0.0:31942, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   41) Thread[id=841, name=qtp240734214-841 Acceptor1 SelectChannelConnector@0.0.0.0:13986, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   42) Thread[id=763, name=qtp1088216030-763 Acceptor3 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   43) Thread[id=605, name=qtp520031133-605 Acceptor2 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   44) Thread[id=630, name=qtp1729456457-630 Acceptor2 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   45) Thread[id=606, name=qtp520031133-606 Acceptor3 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   46) Thread[id=615, name=qtp1296726923-615 Acceptor2 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   47) Thread[id=781, name=qtp1152181387-781 Acceptor0 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   48) Thread[id=639, name=qtp497288441-639 Acceptor1 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   49) Thread[id=842, name=qtp240734214-842 Acceptor2 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   50) Thread[id=613, name=qtp1296726923-613 Acceptor0 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   51) Thread[id=800, name=qtp1294089382-800 Acceptor0 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   52) Thread[id=676, name=qtp804686489-676 Acceptor3 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   53) Thread[id=631, name=qtp1729456457-631 Acceptor3 SelectChannelConnector@0.0.0.0:21124, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   54) Thread[id=771, name=qtp1587559111-771 Acceptor0 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   55) Thread[id=774, name=qtp1587559111-774 Acceptor3 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   56) Thread[id=817, name=qtp95968411-817 Acceptor3 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   57) Thread[id=685, name=qtp1151506535-685 Acceptor0 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   58) Thread[id=761, name=qtp1088216030-761 Acceptor1 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   59) Thread[id=862, name=qtp431075824-862 Acceptor2 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   60) Thread[id=638, name=qtp497288441-638 Acceptor0 SelectChannelConnector@0.0.0.0:36730, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   61) Thread[id=801, name=qtp1294089382-801 Acceptor1 SelectChannelConnector@0.0.0.0:42775, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   62) Thread[id=604, name=qtp520031133-604 Acceptor1 SelectChannelConnector@0.0.0.0:63844, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   63) Thread[id=640, name=qtp497288441-640 Acceptor2 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   64) Thread[id=716, name=qtp1025719152-716 Acceptor1 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=861, name=qtp431075824-861 Acceptor1 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=707, name=qtp1275406235-707 Acceptor2 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=718, name=qtp1025719152-718 Acceptor3 SelectChannelConnector@0.0.0.0:47379, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=686, name=qtp1151506535-686 Acceptor1 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=772, name=qtp1587559111-772 Acceptor1 SelectChannelConnector@0.0.0.0:29265, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=773, name=qtp1587559111-773 Acceptor2 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=840, name=qtp240734214-840 Acceptor0 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=863, name=qtp431075824-863 Acceptor3 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=659, name=qtp983787684-659 Acceptor0 SelectChannelConnector@0.0.0.0:10177, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=661, name=qtp983787684-661 Acceptor2 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=715, name=qtp1025719152-715 Acceptor0 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=816, name=qtp95968411-816 Acceptor2 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=843, name=qtp240734214-843 Acceptor3 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=628, name=qtp1729456457-628 Acceptor0 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=674, name=qtp804686489-674 Acceptor1 SelectChannelConnector@0.0.0.0:32394, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=783, name=qtp1152181387-783 Acceptor2 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=616, name=qtp1296726923-616 Acceptor3 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=629, name=qtp1729456457-629 Acceptor1 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=815, name=qtp95968411-815 Acceptor1 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=603, name=qtp520031133-603 Acceptor0 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=803, name=qtp1294089382-803 Acceptor3 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=687, name=qtp1151506535-687 Acceptor2 SelectChannelConnector@0.0.0.0:49557, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=641, name=qtp497288441-641 Acceptor3 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=760, name=qtp1088216030-760 Acceptor0 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=802, name=qtp1294089382-802 Acceptor2 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=675, name=qtp804686489-675 Acceptor2 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=660, name=qtp983787684-660 Acceptor1 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=662, name=qtp983787684-662 Acceptor3 SelectChannelConnector@0.0.0.0:10177, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=784, name=qtp1152181387-784 Acceptor3 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=762, name=qtp1088216030-762 Acceptor2 SelectChannelConnector@0.0.0.0:16754, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=673, name=qtp804686489-673 Acceptor0 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=614, name=qtp1296726923-614 Acceptor1 SelectChannelConnector@0.0.0.0:34414, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=708, name=qtp1275406235-708 Acceptor3 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=717, name=qtp1025719152-717 Acceptor2 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=782, name=qtp1152181387-782 Acceptor1 SelectChannelConnector@0.0.0.0:25162, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=814, name=qtp95968411-814 Acceptor0 SelectChannelConnector@0.0.0.0:64705, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  37) Thread[id=705, name=qtp1275406235-705 Acceptor0 SelectChannelConnector@0.0.0.0:51732, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  38) Thread[id=688, name=qtp1151506535-688 Acceptor3 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  39) Thread[id=706, name=qtp1275406235-706 Acceptor1 SelectChannelConnector@0.0.0.0:51732, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  40) Thread[id=860, name=qtp431075824-860 Acceptor0 SelectChannelConnector@0.0.0.0:31942, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  41) Thread[id=841, name=qtp240734214-841 Acceptor1 SelectChannelConnector@0.0.0.0:13986, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  42) Thread[id=763, name=qtp1088216030-763 Acceptor3 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  43) Thread[id=605, name=qtp520031133-605 Acceptor2 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  44) Thread[id=630, name=qtp1729456457-630 Acceptor2 SelectChannelConnector@0.0.0.0:21124, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  45) Thread[id=606, name=qtp520031133-606 Acceptor3 SelectChannelConnector@0.0.0.0:63844, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  46) Thread[id=615, name=qtp1296726923-615 Acceptor2 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  47) Thread[id=781, name=qtp1152181387-781 Acceptor0 SelectChannelConnector@0.0.0.0:25162, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  48) Thread[id=639, name=qtp497288441-639 Acceptor1 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  49) Thread[id=842, name=qtp240734214-842 Acceptor2 SelectChannelConnector@0.0.0.0:13986, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  50) Thread[id=613, name=qtp1296726923-613 Acceptor0 SelectChannelConnector@0.0.0.0:34414, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  51) Thread[id=800, name=qtp1294089382-800 Acceptor0 SelectChannelConnector@0.0.0.0:42775, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  52) Thread[id=676, name=qtp804686489-676 Acceptor3 SelectChannelConnector@0.0.0.0:32394, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  53) Thread[id=631, name=qtp1729456457-631 Acceptor3 SelectChannelConnector@0.0.0.0:21124, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  54) Thread[id=771, name=qtp1587559111-771 Acceptor0 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  55) Thread[id=774, name=qtp1587559111-774 Acceptor3 SelectChannelConnector@0.0.0.0:29265, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  56) Thread[id=817, name=qtp95968411-817 Acceptor3 SelectChannelConnector@0.0.0.0:64705, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  57) Thread[id=685, name=qtp1151506535-685 Acceptor0 SelectChannelConnector@0.0.0.0:49557, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  58) Thread[id=761, name=qtp1088216030-761 Acceptor1 SelectChannelConnector@0.0.0.0:16754, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  59) Thread[id=862, name=qtp431075824-862 Acceptor2 SelectChannelConnector@0.0.0.0:31942, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  60) Thread[id=638, name=qtp497288441-638 Acceptor0 SelectChannelConnector@0.0.0.0:36730, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  61) Thread[id=801, name=qtp1294089382-801 Acceptor1 SelectChannelConnector@0.0.0.0:42775, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  62) Thread[id=604, name=qtp520031133-604 Acceptor1 SelectChannelConnector@0.0.0.0:63844, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  63) Thread[id=640, name=qtp497288441-640 Acceptor2 SelectChannelConnector@0.0.0.0:36730, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  64) Thread[id=716, name=qtp1025719152-716 Acceptor1 SelectChannelConnector@0.0.0.0:47379, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([88D599395718F613]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.SolrCmdDistributorTest

Error Message:
20 threads leaked from SUITE scope at org.apache.solr.update.SolrCmdDistributorTest:     1) Thread[id=190, name=qtp1546842717-190 Acceptor3 SelectChannelConnector@0.0.0.0:48886, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=189, name=qtp1546842717-189 Acceptor2 SelectChannelConnector@0.0.0.0:48886, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=201, name=qtp957277542-201 Acceptor1 SelectChannelConnector@0.0.0.0:20620, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=148, name=qtp629745675-148 Acceptor3 SelectChannelConnector@0.0.0.0:57594, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=188, name=qtp1546842717-188 Acceptor1 SelectChannelConnector@0.0.0.0:48886, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=159, name=qtp54809062-159 Acceptor1 SelectChannelConnector@0.0.0.0:23275, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=202, name=qtp957277542-202 Acceptor2 SelectChannelConnector@0.0.0.0:20620, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=187, name=qtp1546842717-187 Acceptor0 SelectChannelConnector@0.0.0.0:48886, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=173, name=qtp418705882-173 Acceptor0 SelectChannelConnector@0.0.0.0:42905, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=158, name=qtp54809062-158 Acceptor0 SelectChannelConnector@0.0.0.0:23275, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=161, name=qtp54809062-161 Acceptor3 SelectChannelConnector@0.0.0.0:23275, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=160, name=qtp54809062-160 Acceptor2 SelectChannelConnector@0.0.0.0:23275, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=146, name=qtp629745675-146 Acceptor1 SelectChannelConnector@0.0.0.0:57594, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=145, name=qtp629745675-145 Acceptor0 SelectChannelConnector@0.0.0.0:57594, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=176, name=qtp418705882-176 Acceptor3 SelectChannelConnector@0.0.0.0:42905, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=203, name=qtp957277542-203 Acceptor3 SelectChannelConnector@0.0.0.0:20620, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=175, name=qtp418705882-175 Acceptor2 SelectChannelConnector@0.0.0.0:42905, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=200, name=qtp957277542-200 Acceptor0 SelectChannelConnector@0.0.0.0:20620, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=147, name=qtp629745675-147 Acceptor2 SelectChannelConnector@0.0.0.0:57594, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=174, name=qtp418705882-174 Acceptor1 SelectChannelConnector@0.0.0.0:42905, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 20 threads leaked from SUITE scope at org.apache.solr.update.SolrCmdDistributorTest: 
   1) Thread[id=190, name=qtp1546842717-190 Acceptor3 SelectChannelConnector@0.0.0.0:48886, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=189, name=qtp1546842717-189 Acceptor2 SelectChannelConnector@0.0.0.0:48886, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=201, name=qtp957277542-201 Acceptor1 SelectChannelConnector@0.0.0.0:20620, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=148, name=qtp629745675-148 Acceptor3 SelectChannelConnector@0.0.0.0:57594, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector

[...truncated too long message...]

 org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=161, name=qtp54809062-161 Acceptor3 SelectChannelConnector@0.0.0.0:23275, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=160, name=qtp54809062-160 Acceptor2 SelectChannelConnector@0.0.0.0:23275, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=146, name=qtp629745675-146 Acceptor1 SelectChannelConnector@0.0.0.0:57594, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=145, name=qtp629745675-145 Acceptor0 SelectChannelConnector@0.0.0.0:57594, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=176, name=qtp418705882-176 Acceptor3 SelectChannelConnector@0.0.0.0:42905, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=203, name=qtp957277542-203 Acceptor3 SelectChannelConnector@0.0.0.0:20620, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   17) Thread[id=175, name=qtp418705882-175 Acceptor2 SelectChannelConnector@0.0.0.0:42905, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   18) Thread[id=200, name=qtp957277542-200 Acceptor0 SelectChannelConnector@0.0.0.0:20620, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   19) Thread[id=147, name=qtp629745675-147 Acceptor2 SelectChannelConnector@0.0.0.0:57594, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   20) Thread[id=174, name=qtp418705882-174 Acceptor1 SelectChannelConnector@0.0.0.0:42905, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([88D599395718F613]:0)
[junit4:junit4] Completed on J0 in 30.72s, 1 test, 2 errors <<< FAILURES!

[...truncated 146 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 780 ignored

Total time: 26 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 834 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/834/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=347, name=qtp1954477215-347 Acceptor2 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=395, name=qtp1602961426-395 Acceptor1 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=374, name=qtp1219785243-374 Acceptor0 SelectChannelConnector@0.0.0.0:19621, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=345, name=qtp1954477215-345 Acceptor0 SelectChannelConnector@0.0.0.0:33948, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=376, name=qtp1219785243-376 Acceptor2 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=394, name=qtp1602961426-394 Acceptor0 SelectChannelConnector@0.0.0.0:55508, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=348, name=qtp1954477215-348 Acceptor3 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=397, name=qtp1602961426-397 Acceptor3 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=375, name=qtp1219785243-375 Acceptor1 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=396, name=qtp1602961426-396 Acceptor2 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=346, name=qtp1954477215-346 Acceptor1 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=377, name=qtp1219785243-377 Acceptor3 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=347, name=qtp1954477215-347 Acceptor2 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=395, name=qtp1602961426-395 Acceptor1 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=374, name=qtp1219785243-374 Acceptor0 SelectChannelConnector@0.0.0.0:19621, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=345, name=qtp1954477215-345 Acceptor0 SelectChannelConnector@0.0.0.0:33948, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=376, name=qtp1219785243-376 Acceptor2 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=394, name=qtp1602961426-394 Acceptor0 SelectChannelConnector@0.0.0.0:55508, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=348, name=qtp1954477215-348 Acceptor3 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=397, name=qtp1602961426-397 Acceptor3 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=375, name=qtp1219785243-375 Acceptor1 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=396, name=qtp1602961426-396 Acceptor2 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=346, name=qtp1954477215-346 Acceptor1 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=377, name=qtp1219785243-377 Acceptor3 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([F31EC422C651F5E9]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=347, name=qtp1954477215-347 Acceptor2 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=375, name=qtp1219785243-375 Acceptor1 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=395, name=qtp1602961426-395 Acceptor1 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=374, name=qtp1219785243-374 Acceptor0 SelectChannelConnector@0.0.0.0:19621, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=345, name=qtp1954477215-345 Acceptor0 SelectChannelConnector@0.0.0.0:33948, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=376, name=qtp1219785243-376 Acceptor2 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=396, name=qtp1602961426-396 Acceptor2 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=346, name=qtp1954477215-346 Acceptor1 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=377, name=qtp1219785243-377 Acceptor3 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=394, name=qtp1602961426-394 Acceptor0 SelectChannelConnector@0.0.0.0:55508, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=348, name=qtp1954477215-348 Acceptor3 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=397, name=qtp1602961426-397 Acceptor3 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=347, name=qtp1954477215-347 Acceptor2 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=375, name=qtp1219785243-375 Acceptor1 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=395, name=qtp1602961426-395 Acceptor1 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=374, name=qtp1219785243-374 Acceptor0 SelectChannelConnector@0.0.0.0:19621, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=345, name=qtp1954477215-345 Acceptor0 SelectChannelConnector@0.0.0.0:33948, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=376, name=qtp1219785243-376 Acceptor2 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=396, name=qtp1602961426-396 Acceptor2 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=346, name=qtp1954477215-346 Acceptor1 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=377, name=qtp1219785243-377 Acceptor3 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=394, name=qtp1602961426-394 Acceptor0 SelectChannelConnector@0.0.0.0:55508, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=348, name=qtp1954477215-348 Acceptor3 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=397, name=qtp1602961426-397 Acceptor3 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([F31EC422C651F5E9]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedQueryElevationComponentTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedQueryElevationComponentTest:     1) Thread[id=274, name=qtp898671304-274 Acceptor2 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=251, name=qtp1029168456-251 Acceptor3 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=234, name=qtp436463023-234 Acceptor0 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=272, name=qtp898671304-272 Acceptor0 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=235, name=qtp436463023-235 Acceptor1 SelectChannelConnector@0.0.0.0:11830, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=258, name=qtp868366762-258 Acceptor0 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=237, name=qtp436463023-237 Acceptor3 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=250, name=qtp1029168456-250 Acceptor2 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=248, name=qtp1029168456-248 Acceptor0 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=236, name=qtp436463023-236 Acceptor2 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=249, name=qtp1029168456-249 Acceptor1 SelectChannelConnector@0.0.0.0:36779, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=259, name=qtp868366762-259 Acceptor1 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=261, name=qtp868366762-261 Acceptor3 SelectChannelConnector@0.0.0.0:65378, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=260, name=qtp868366762-260 Acceptor2 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=273, name=qtp898671304-273 Acceptor1 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=275, name=qtp898671304-275 Acceptor3 SelectChannelConnector@0.0.0.0:53313, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedQueryElevationComponentTest: 
   1) Thread[id=274, name=qtp898671304-274 Acceptor2 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=251, name=qtp1029168456-251 Acceptor3 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=234, name=qtp436463023-234 Acceptor0 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=272, name=qtp898671304-272 Acceptor0 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=235, name=qtp436463023-235 Acceptor1 SelectChannelConnector@0.0.0.0:11830, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=258, name=qtp868366762-258 Acceptor0 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=237, name=qtp436463023-237 Acceptor3 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=250, name=qtp1029168456-250 Acceptor2 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=248, name=qtp1029168456-248 Acceptor0 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=236, name=qtp436463023-236 Acceptor2 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=249, name=qtp1029168456-249 Acceptor1 SelectChannelConnector@0.0.0.0:36779, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=259, name=qtp868366762-259 Acceptor1 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=261, name=qtp868366762-261 Acceptor3 SelectChannelConnector@0.0.0.0:65378, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=260, name=qtp868366762-260 Acceptor2 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=273, name=qtp898671304-273 Acceptor1 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=275, name=qtp898671304-275 Acceptor3 SelectChannelConnector@0.0.0.0:53313, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([F31EC422C651F5E9]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedQueryElevationComponentTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=251, name=qtp1029168456-251 Acceptor3 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=274, name=qtp898671304-274 Acceptor2 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=234, name=qtp436463023-234 Acceptor0 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=272, name=qtp898671304-272 Acceptor0 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=235, name=qtp436463023-235 Acceptor1 SelectChannelConnector@0.0.0.0:11830, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=258, name=qtp868366762-258 Acceptor0 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=237, name=qtp436463023-237 Acceptor3 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=236, name=qtp436463023-236 Acceptor2 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=248, name=qtp1029168456-248 Acceptor0 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=250, name=qtp1029168456-250 Acceptor2 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=249, name=qtp1029168456-249 Acceptor1 SelectChannelConnector@0.0.0.0:36779, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=259, name=qtp868366762-259 Acceptor1 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=261, name=qtp868366762-261 Acceptor3 SelectChannelConnector@0.0.0.0:65378, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=260, name=qtp868366762-260 Acceptor2 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=273, name=qtp898671304-273 Acceptor1 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=275, name=qtp898671304-275 Acceptor3 SelectChannelConnector@0.0.0.0:53313, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=251, name=qtp1029168456-251 Acceptor3 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=274, name=qtp898671304-274 Acceptor2 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=234, name=qtp436463023-234 Acceptor0 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=272, name=qtp898671304-272 Acceptor0 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=235, name=qtp436463023-235 Acceptor1 SelectChannelConnector@0.0.0.0:11830, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=258, name=qtp868366762-258 Acceptor0 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=237, name=qtp436463023-237 Acceptor3 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=236, name=qtp436463023-236 Acceptor2 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=248, name=qtp1029168456-248 Acceptor0 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=250, name=qtp1029168456-250 Acceptor2 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=249, name=qtp1029168456-249 Acceptor1 SelectChannelConnector@0.0.0.0:36779, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=259, name=qtp868366762-259 Acceptor1 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=261, name=qtp868366762-261 Acceptor3 SelectChannelConnector@0.0.0.0:65378, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=260, name=qtp868366762-260 Acceptor2 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=273, name=qtp898671304-273 Acceptor1 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=275, name=qtp898671304-275 Acceptor3 SelectChannelConnector@0.0.0.0:53313, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([F31EC422C651F5E9]:0)




Build Log:
[...truncated 7717 lines...]
[junit4:junit4] Suite: org.apache.solr.handler.component.DistributedQueryElevationComponentTest
[junit4:junit4]   2> 4 T227 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 538 T227 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 826 T227 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:11830
[junit4:junit4]   2> 854 T227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 854 T227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 854 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 887 T227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 887 T227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 887 T227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 888 T227 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 888 T227 oasc.CoreContainer.<init> New CoreContainer 1158879031
[junit4:junit4]   2> 889 T227 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 889 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 913 T227 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 960 T227 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 961 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 962 T227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 962 T227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 994 T227 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1036 T227 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-elevate.xml
[junit4:junit4]   2> 1037 T227 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1042 T227 oass.IndexSchema.readSchema Schema name=example
[junit4:junit4]   2> 1223 T227 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1499 T227 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1513 T227 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/control/data/
[junit4:junit4]   2> 1514 T227 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@26b496d
[junit4:junit4]   2> 1515 T227 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/control/data/index/
[junit4:junit4]   2> 1516 T227 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1517 T227 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/control/data/index forceNew:false
[junit4:junit4]   2> 1518 T227 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d662625 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@753af01f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1518 T227 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1535 T227 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1535 T227 oasc.RequestHandlers.initHandlersFromConfig created /elevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1536 T227 oasc.RequestHandlers.initHandlersFromConfig created /dataElevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1541 T227 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1542 T227 oasc.RequestHandlers.initHandlersFromConfig WARNING no default request handler is registered (either '/select' or 'standard')
[junit4:junit4]   2> 1543 T227 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 1545 T227 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-MINIMAL.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-MERGEPOLICY.XML, CURRENCY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, SCHEMA-REVERSED.XML, STOPTYPES-2.TXT, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SOLRCONFIG-PHRASESUGGEST.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 1547 T227 oass.SolrIndexSearcher.<init> Opening Searcher@5d884c5a main
[junit4:junit4]   2> 1547 T227 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1548 T227 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1548 T227 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1560 T227 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 1574 T227 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 1583 T227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1584 T227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1584 T227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1585 T227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1585 T227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1585 T227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1586 T227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1586 T227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1586 T227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1587 T227 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1599 T242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5d884c5a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1600 T227 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1601 T227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 1601 T227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1644 T227 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1661 T227 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1663 T227 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:36779
[junit4:junit4]   2> 1664 T227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1664 T227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1665 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1688 T227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1688 T227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1688 T227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1689 T227 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1689 T227 oasc.CoreContainer.<init> New CoreContainer 895344114
[junit4:junit4]   2> 1690 T227 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1690 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1706 T227 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1719 T227 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1720 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1721 T227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1721 T227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1753 T227 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1792 T227 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-elevate.xml
[junit4:junit4]   2> 1793 T227 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1797 T227 oass.IndexSchema.readSchema Schema name=example
[junit4:junit4]   2> 1964 T227 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1967 T227 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1974 T227 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard0/data/
[junit4:junit4]   2> 1975 T227 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@26b496d
[junit4:junit4]   2> 1976 T227 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard0/data/index/
[junit4:junit4]   2> 1977 T227 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1977 T227 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard0/data/index forceNew:false
[junit4:junit4]   2> 1978 T227 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1362cffa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20ad3a7a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1979 T227 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1981 T227 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1982 T227 oasc.RequestHandlers.initHandlersFromConfig created /elevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1982 T227 oasc.RequestHandlers.initHandlersFromConfig created /dataElevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1988 T227 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1989 T227 oasc.RequestHandlers.initHandlersFromConfig WARNING no default request handler is registered (either '/select' or 'standard')
[junit4:junit4]   2> 1990 T227 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 1991 T227 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-MINIMAL.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-MERGEPOLICY.XML, CURRENCY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, SCHEMA-REVERSED.XML, STOPTYPES-2.TXT, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SOLRCONFIG-PHRASESUGGEST.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 1993 T227 oass.SolrIndexSearcher.<init> Opening Searcher@6a51c5bd main
[junit4:junit4]   2> 1993 T227 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1994 T227 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1994 T227 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1995 T227 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 2005 T227 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 2015 T227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2016 T227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2016 T227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2016 T227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2017 T227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2017 T227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2017 T227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2017 T227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2018 T227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2018 T227 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2032 T252 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6a51c5bd main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2033 T227 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2034 T227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2034 T227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2034 T227 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2043 T227 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2045 T227 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:65378
[junit4:junit4]   2> 2045 T227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2046 T227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2046 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2058 T227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2058 T227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2058 T227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2059 T227 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2059 T227 oasc.CoreContainer.<init> New CoreContainer 71827195
[junit4:junit4]   2> 2059 T227 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2060 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2075 T227 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2089 T227 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2089 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2090 T227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2090 T227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2120 T227 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2157 T227 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-elevate.xml
[junit4:junit4]   2> 2158 T227 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2163 T227 oass.IndexSchema.readSchema Schema name=example
[junit4:junit4]   2> 2317 T227 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2319 T227 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2326 T227 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard1/data/
[junit4:junit4]   2> 2326 T227 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@26b496d
[junit4:junit4]   2> 2327 T227 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard1/data/index/
[junit4:junit4]   2> 2328 T227 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2328 T227 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard1/data/index forceNew:false
[junit4:junit4]   2> 2329 T227 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3cbbad9a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@78bc6d6a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2330 T227 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2332 T227 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2332 T227 oasc.RequestHandlers.initHandlersFromConfig created /elevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2333 T227 oasc.RequestHandlers.initHandlersFromConfig created /dataElevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2334 T227 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2335 T227 oasc.RequestHandlers.initHandlersFromConfig WARNING no default request handler is registered (either '/select' or 'standard')
[junit4:junit4]   2> 2336 T227 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 2337 T227 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-MINIMAL.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-MERGEPOLICY.XML, CURRENCY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, SCHEMA-REVERSED.XML, STOPTYPES-2.TXT, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SOLRCONFIG-PHRASESUGGEST.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 2339 T227 oass.SolrIndexSearcher.<init> Opening Searcher@4e931efa main
[junit4:junit4]   2> 2339 T227 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 2340 T227 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2340 T227 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2341 T227 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 2351 T227 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 2360 T227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2361 T227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2361 T227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2361 T227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2362 T227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2362 T227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2362 T227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2362 T227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2363 T227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2363 T227 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2376 T227 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2377 T227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2378 T227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2378 T227 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2380 T266 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4e931efa main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2392 T227 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2396 T227 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:53313
[junit4:junit4]   2> 2397 T227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2398 T227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2398 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2416 T227 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2417 T227 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2417 T227 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2418 T227 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2418 T227 oasc.CoreContainer.<init> New CoreContainer 1799397986
[junit4:junit4]   2> 2419 T227 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2419 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2442 T227 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2464 T227 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2465 T227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2466 T227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2467 T227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2511 T227 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2569 T227 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-elevate.xml
[junit4:junit4]   2> 2570 T227 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2577 T227 oass.IndexSchema.readSchema Schema name=example
[junit4:junit4]   2> 2825 T227 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2827 T227 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2836 T227 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard2/data/
[junit4:junit4]   2> 2836 T227 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@26b496d
[junit4:junit4]   2> 2837 T227 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard2/data/index/
[junit4:junit4]   2> 2838 T227 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2991 T227 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351957282878/shard2/data/index forceNew:false
[junit4:junit4]   2> 2993 T227 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61d10846 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@490a7d73),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2993 T227 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2996 T227 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2997 T227 oasc.RequestHandlers.initHandlersFromConfig created /elevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2997 T227 oasc.RequestHandlers.initHandlersFromConfig created /dataElevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2999 T227 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2999 T227 oasc.RequestHandlers.initHandlersFromConfig WARNING no default request handler is registered (either '/select' or 'standard')
[junit4:junit4]   2> 3001 T227 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 3003 T227 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-MINIMAL.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-MERGEPOLICY.XML, CURRENCY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, SCHEMA-REVERSED.XML, STOPTYPES-2.TXT, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SOLRCONFIG-PHRASESUGGEST.XML, SCHEMA-DFR.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, SCHEMA12.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 3019 T227 oass.SolrIndexSearcher.<init> Opening Searcher@37a417e7 main
[junit4:junit4]   2> 3019 T227 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3020 T227 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 3021 T227 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 3022 T227 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 3037 T227 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 3051 T227 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 3051 T227 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3052 T227 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 3052 T227 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3052 T227 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3053 T227 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3053 T227 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3053 T227 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3054 T227 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3054 T227 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 3121 T282 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37a417e7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 3122 T227 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 3123 T227 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3123 T227 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3123 T227 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C17 name=collection1 org.apache.solr.core.SolrCore@131feb95
[junit4:junit4]   2> 4565 T239 C17 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 4568 T239 C17 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d662625 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@753af01f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4568 T239 C17 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4569 T239 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 25
[junit4:junit4]   2> ASYNC  NEW_CORE C18 name=collection1 org.apache.solr.core.SolrCore@39b2f2ce
[junit4:junit4]   2> 4727 T263 C18 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 4730 T263 C18 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1362cffa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20ad3a7a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4731 T263 C18 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4731 T263 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 4
[junit4:junit4]   2> ASYNC  NEW_CORE C19 name=collection1 org.apache.solr.core.SolrCore@63dceace
[junit4:junit4]   2> 4818 T277 C19 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 4903 T277 C19 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3cbbad9a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@78bc6d6a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4903 T277 C19 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4904 T277 C19 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 86
[junit4:junit4]   2> ASYNC  NEW_CORE C20 name=collection1 org.apache.solr.core.SolrCore@1c1e9a0
[junit4:junit4]   2> 4912 T278 C20 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 4915 T278 C20 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61d10846 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@490a7d73),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4915 T278 C20 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4916 T278 C20 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 4
[junit4:junit4]   2> 4922 T241 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 2
[junit4:junit4]   2> 4927 T265 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 1
[junit4:junit4]   2> 4932 T239 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 2
[junit4:junit4]   2> 4937 T283 C19 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 2
[junit4:junit4]   2> 4941 T240 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 1
[junit4:junit4]   2> 4946 T280 C20 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 1
[junit4:junit4]   2> 4950 T241 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 1
[junit4:junit4]   2> 4955 T263 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 2
[junit4:junit4]   2> 4959 T239 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 1
[junit4:junit4]   2> 4964 T284 C19 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 1
[junit4:junit4]   2> 4968 T240 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 0
[junit4:junit4]   2> 4976 T281 C20 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 2
[junit4:junit4]   2> 4981 T241 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 0
[junit4:junit4]   2> 4986 T278 C20 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 2
[junit4:junit4]   2> 5020 T239 C17 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5023 T239 C17 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d662625 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@753af01f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7d662625 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@753af01f),segFN=segments_2,generation=2,filenames=[_0_Asserting_0.tim, _0.fnm, _0_Asserting_0.tip, _0_Memory_0.ram, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Asserting_0.pos, _0.fdt, _0_Asserting_0.doc]
[junit4:junit4]   2> 5024 T239 C17 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 5026 T239 C17 oass.SolrIndexSearcher.<init> Opening Searcher@13371dc8 main
[junit4:junit4]   2> 5026 T239 C17 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5028 T239 C17 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5029 T242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13371dc8 main{StandardDirectoryReader(segments_2:4 _0(4.1):C7)}
[junit4:junit4]   2> 5030 T239 C17 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 10
[junit4:junit4]   2> 5034 T264 C18 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5036 T264 C18 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1362cffa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20ad3a7a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1362cffa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@20ad3a7a),segFN=segments_2,generation=2,filenames=[_0_Asserting_0.tim, _0.fnm, _0_Asserting_0.tip, _0_Memory_0.ram, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Asserting_0.pos, _0.fdt, _0_Asserting_0.doc]
[junit4:junit4]   2> 5036 T264 C18 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 5038 T264 C18 oass.SolrIndexSearcher.<init> Opening Searcher@102bdf72 main
[junit4:junit4]   2> 5039 T264 C18 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5039 T264 C18 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5041 T252 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@102bdf72 main{StandardDirectoryReader(segments_2:4 _0(4.1):C2)}
[junit4:junit4]   2> 5042 T264 C18 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 8
[junit4:junit4]   2> 5044 T277 C19 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5046 T277 C19 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3cbbad9a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@78bc6d6a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3cbbad9a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@78bc6d6a),segFN=segments_2,generation=2,filenames=[_0_Asserting_0.tim, _0.fnm, _0_Asserting_0.tip, _0_Memory_0.ram, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Asserting_0.pos, _0.fdt, _0_Asserting_0.doc]
[junit4:junit4]   2> 5047 T277 C19 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 5049 T277 C19 oass.SolrIndexSearcher.<init> Opening Searcher@3e872ac6 main
[junit4:junit4]   2> 5049 T277 C19 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5050 T277 C19 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5052 T266 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3e872ac6 main{StandardDirectoryReader(segments_2:4 _0(4.1):C2)}
[junit4:junit4]   2> 5053 T277 C19 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 9
[junit4:junit4]   2> 5056 T280 C20 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5058 T280 C20 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61d10846 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@490a7d73),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61d10846 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@490a7d73),segFN=segments_2,generation=2,filenames=[_0_Asserting_0.tim, _0.fnm, _0_Asserting_0.tip, _0_Memory_0.ram, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Asserting_0.pos, _0.fdt, _0_Asserting_0.doc]
[junit4:junit4]   2> 5058 T280 C20 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 5060 T280 C20 oass.SolrIndexSearcher.<init> Opening Searcher@621cc66c main
[junit4:junit4]   2> 5060 T280 C20 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5061 T280 C20 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5064 T282 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@621cc66c main{StandardDirectoryReader(segments_2:4 _0(4.1):C3)}
[junit4:junit4]   2> 5064 T280 C20 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 8
[junit4:junit4]   2> 5164 T240 C17 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&fl=id,+score,+[elevated]&shards.qt=/elevate&q=*:*&distrib=false&qt=/elevate&wt=javabin&rows=500&version=2} hits=7 status=0 QTime=57 
[junit4:junit4]   2> 5247 T285 C19 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351957288043&shard.url=[ff01::114]:33332/solr|127.0.0.1:65378/solr|[ff01::083]:33332/solr&fl=id,score&start=0&q=*:*&isShard=true&qt=/elevate&fsv=true} hits=2 status=0 QTime=11 
[junit4:junit4]   2> 5251 T265 C18 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351957288043&shard.url=127.0.0.1:36779/solr|[ff01::083]:33332/solr&fl=id,score&start=0&q=*:*&isShard=true&qt=/elevate&fsv=true} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 5253 T278 C20 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351957288043&shard.url=[ff01::083]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:53313/solr&fl=id,score&start=0&q=*:*&isShard=true&qt=/elevate&fsv=true} hits=3 status=0 QTime=1 
[junit4:junit4]   2> 5283 T263 C18 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=4.0,1.0&distrib=false&wt=javabin&rows=500&version=2&fl=id,+score,+[elevated]&shard.url=127.0.0.1:36779/solr|[ff01::083]:33332/solr&NOW=1351957288043&q=*:*&qt=/elevate&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 5283 T283 C19 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=5.0,2.0&distrib=false&wt=javabin&rows=500&version=2&fl=id,+score,+[elevated]&shard.url=[ff01::114]:33332/solr|127.0.0.1:65378/solr|[ff01::083]:33332/solr&NOW=1351957288043&q=*:*&qt=/elevate&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 5283 T280 C20 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=3.0,6.0,7.0&distrib=false&wt=javabin&rows=500&version=2&fl=id,+score,+[elevated]&shard.url=[ff01::083]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:53313/solr&NOW=1351957288043&q=*:*&qt=/elevate&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 5285 T281 C20 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&fl=id,+score,+[elevated]&shards=127.0.0.1:36779/solr|[ff01::083]:33332/solr,[ff01::114]:33332/solr|127.0.0.1:65378/solr|[ff01::083]:33332/solr,[ff01::083]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:53313/solr&shards.qt=/elevate&q=*:*&qt=/elevate&wt=javabin&rows=500&version=2} hits=7 status=0 QTime=116 
[junit4:junit4]   2> 5306 T239 C17 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=int_i+desc&shards.qt=/elevate&q=ZZZZ&forceElevation=true&distrib=false&qt=/elevate&wt=javabin&rows=500&version=2} hits=5 status=0 QTime=14 
[junit4:junit4]   2> 5319 T265 C18 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351957288185&shard.url=127.0.0.1:36779/solr&fl=id&start=0&q=ZZZZ&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=1 status=0 QTime=2 
[junit4:junit4]   2> 5320 T281 C20 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351957288185&shard.url=127.0.0.1:53313/solr|[ff01::213]:33332/solr&fl=id&start=0&q=ZZZZ&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 5325 T285 C19 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351957288185&shard.url=[ff01::114]:33332/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:65378/solr&fl=id&start=0&q=ZZZZ&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=2 status=0 QTime=2 
[junit4:junit4]   2> 5332 T283 C19 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=5.0,2.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=[ff01::114]:33332/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:65378/solr&NOW=1351957288185&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 5332 T264 C18 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=1.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=127.0.0.1:36779/solr&NOW=1351957288185&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=1 
[junit4:junit4]   2> 5332 T278 C20 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=3.0,6.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=127.0.0.1:53313/solr|[ff01::213]:33332/solr&NOW=1351957288185&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=1 
[junit4:junit4]   2> 5334 T263 C18 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=int_i+desc&shards=127.0.0.1:36779/solr,[ff01::114]:33332/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:65378/solr,127.0.0.1:53313/solr|[ff01::213]:33332/solr&shards.qt=/elevate&q=ZZZZ&forceElevation=true&qt=/elevate&wt=javabin&rows=500&version=2} hits=5 status=0 QTime=23 
[junit4:junit4]   2> 5339 T241 C17 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=int_i+asc&shards.qt=/elevate&q=solr&forceElevation=true&distrib=false&qt=/elevate&wt=javabin&rows=500&version=2} hits=1 status=0 QTime=2 
[junit4:junit4]   2> 5352 T264 C18 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+asc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351957288217&shard.url=127.0.0.1:36779/solr&fl=id&start=0&q=solr&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 5355 T285 C19 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+asc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351957288217&shard.url=127.0.0.1:65378/solr|[ff01::114]:33332/solr|[ff01::213]:33332/solr&fl=id&start=0&q=solr&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 5357 T280 C20 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+asc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351957288217&shard.url=[ff01::213]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:53313/solr&fl=id&start=0&q=solr&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=1 status=0 QTime=4 
[junit4:junit4]   2> 5375 T281 C20 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=7.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=[ff01::213]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:53313/solr&NOW=1351957288217&q=solr&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 5377 T284 C19 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=int_i+asc&shards=127.0.0.1:36779/solr,127.0.0.1:65378/solr|[ff01::114]:33332/solr|[ff01::213]:33332/solr,[ff01::213]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:53313/solr&shards.qt=/elevate&q=solr&forceElevation=true&qt=/elevate&wt=javabin&rows=500&version=2} hits=1 status=0 QTime=34 
[junit4:junit4]   2> 5383 T239 C17 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=id+desc&shards.qt=/elevate&q=ZZZZ&forceElevation=true&distrib=false&qt=/elevate&wt=javabin&rows=500&version=2} hits=5 status=0 QTime=2 
[junit4:junit4]   2> 5393 T265 C18 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&fl=id&shard.url=127.0.0.1:36779/solr|[ff01::213]:33332/solr&NOW=1351957288261&start=0&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true&fsv=true} hits=1 status=0 QTime=2 
[junit4:junit4]   2> 5393 T283 C19 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&fl=id&shard.url=127.0.0.1:65378/solr|[ff01::083]:33332/solr&NOW=1351957288261&start=0&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true&fsv=true} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 5393 T278 C20 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&fl=id&shard.url=[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:53313/solr|[ff01::114]:33332/solr&NOW=1351957288261&start=0&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true&fsv=true} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 5398 T264 C18 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=1.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=127.0.0.1:36779/solr|[ff01::213]:33332/solr&NOW=1351957288261&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 5399 T280 C20 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=3.0,6.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:53313/solr|[ff01::114]:33332/solr&NOW=1351957288261&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=1 
[junit4:junit4]   2> 5401 T277 C19 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=5.0,2.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=127.0.0.1:65378/solr|[ff01::083]:33332/solr&NOW=1351957288261&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=3 
[junit4:junit4]   2> 5403 T263 C18 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=id+desc&shards=127.0.0.1:36779/solr|[ff01::213]:33332/solr,127.0.0.1:65378/solr|[ff01::083]:33332/solr,[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:53313/solr|[ff01::114]:33332/solr&shards.qt=/elevate&q=ZZZZ&forceElevation=true&qt=/elevate&wt=javabin&rows=500&version=2} hits=5 status=0 QTime=16 
[junit4:junit4]   2> 5451 T227 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1158879031
[junit4:junit4]   2> 5451 T227 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@131feb95
[junit4:junit4]   2> 5455 T227 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=7,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 5456 T227 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 5457 T227 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 5457 T227 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 5459 T227 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 5463 T227 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 5563 T227 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 5613 T227 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=895344114
[junit4:junit4]   2> 5613 T227 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@39b2f2ce
[junit4:junit4]   2> 5616 T227 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 5617 T227 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 5618 T227 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 5618 T227 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 5619 T227 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 5622 T227 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 5723 T227 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 5759 T227 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=71827195
[junit4:junit4]   2> 5760 T227 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@63dceace
[junit4:junit4]   2> 5763 T227 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 5764 T227 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 5765 T227 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 5765 T227 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 5767 T227 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 5771 T227 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 5871 T227 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 5916 T227 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1799397986
[junit4:junit4]   2> 5917 T227 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1c1e9a0
[junit4:junit4]   2> 5923 T227 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 5924 T227 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 5924 T227 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 5925 T227 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 5926 T227 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 5929 T227 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 6030 T227 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 6031 T227 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 6034 T227 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 6040 T226 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 16 leaked thread(s).
[junit4:junit4]   2> 26154 T226 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 16 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedQueryElevationComponentTest: 
[junit4:junit4]   2> 	   1) Thread[id=274, name=qtp898671304-274 Acceptor2 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=251, name=qtp1029168456-251 Acceptor3 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=234, name=qtp436463023-234 Acceptor0 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=272, name=qtp898671304-272 Acceptor0 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   5) Thread[id=235, name=qtp436463023-235 Acceptor1 SelectChannelConnector@0.0.0.0:11830, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   6) Thread[id=258, name=qtp868366762-258 Acceptor0 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   7) Thread[id=237, name=qtp436463023-237 Acceptor3 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   8) Thread[id=250, name=qtp1029168456-250 Acceptor2 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   9) Thread[id=248, name=qtp1029168456-248 Acceptor0 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  10) Thread[id=236, name=qtp436463023-236 Acceptor2 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  11) Thread[id=249, name=qtp1029168456-249 Acceptor1 SelectChannelConnector@0.0.0.0:36779, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  12) Thread[id=259, name=qtp868366762-259 Acceptor1 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  13) Thread[id=261, name=qtp868366762-261 Acceptor3 SelectChannelConnector@0.0.0.0:65378, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  14) Thread[id=260, name=qtp868366762-260 Acceptor2 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  15) Thread[id=273, name=qtp898671304-273 Acceptor1 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  16) Thread[id=275, name=qtp898671304-275 Acceptor3 SelectChannelConnector@0.0.0.0:53313, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 26156 T226 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=274, name=qtp898671304-274 Acceptor2 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   2) Thread[id=251, name=qtp1029168456-251 Acceptor3 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   3) Thread[id=234, name=qtp436463023-234 Acceptor0 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   4) Thread[id=272, name=qtp898671304-272 Acceptor0 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   5) Thread[id=235, name=qtp436463023-235 Acceptor1 SelectChannelConnector@0.0.0.0:11830, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   6) Thread[id=258, name=qtp868366762-258 Acceptor0 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   7) Thread[id=237, name=qtp436463023-237 Acceptor3 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   8) Thread[id=250, name=qtp1029168456-250 Acceptor2 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   9) Thread[id=248, name=qtp1029168456-248 Acceptor0 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  10) Thread[id=236, name=qtp436463023-236 Acceptor2 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  11) Thread[id=249, name=qtp1029168456-249 Acceptor1 SelectChannelConnector@0.0.0.0:36779, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  12) Thread[id=259, name=qtp868366762-259 Acceptor1 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  13) Thread[id=261, name=qtp868366762-261 Acceptor3 SelectChannelConnector@0.0.0.0:65378, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  14) Thread[id=260, name=qtp868366762-260 Acceptor2 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  15) Thread[id=273, name=qtp898671304-273 Acceptor1 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  16) Thread[id=275, name=qtp898671304-275 Acceptor3 SelectChannelConnector@0.0.0.0:53313, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 29346 T226 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=251, name=qtp1029168456-251 Acceptor3 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=274, name=qtp898671304-274 Acceptor2 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=234, name=qtp436463023-234 Acceptor0 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=272, name=qtp898671304-272 Acceptor0 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   5) Thread[id=235, name=qtp436463023-235 Acceptor1 SelectChannelConnector@0.0.0.0:11830, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   6) Thread[id=258, name=qtp868366762-258 Acceptor0 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   7) Thread[id=237, name=qtp436463023-237 Acceptor3 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   8) Thread[id=236, name=qtp436463023-236 Acceptor2 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   9) Thread[id=248, name=qtp1029168456-248 Acceptor0 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  10) Thread[id=250, name=qtp1029168456-250 Acceptor2 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  11) Thread[id=249, name=qtp1029168456-249 Acceptor1 SelectChannelConnector@0.0.0.0:36779, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  12) Thread[id=259, name=qtp868366762-259 Acceptor1 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  13) Thread[id=261, name=qtp868366762-261 Acceptor3 SelectChannelConnector@0.0.0.0:65378, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  14) Thread[id=260, name=qtp868366762-260 Acceptor2 SelectChannelConnector@0.0.0.0:65378, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  15) Thread[id=273, name=qtp898671304-273 Acceptor1 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  16) Thread[id=275, name=qtp898671304-275 Acceptor3 SelectChannelConnector@0.0.0.0:53313, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {plow_t=PostingsFormat(name=Asserting), id=PostingsFormat(name=Memory doPackFST= false), field_t=PostingsFormat(name=Memory doPackFST= false), text=PostingsFormat(name=Asserting), int_i=PostingsFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=th_TH, timezone=Australia/LHI
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=17,free=346962304,total=502726656
[junit4:junit4]   2> NOTE: All tests run in this JVM: [SpellingQueryConverterTest, TestSolrDeletionPolicy2, SuggesterTest, SoftAutoCommitTest, SignatureUpdateProcessorFactoryTest, TestLRUCache, TestAnalyzedSuggestions, OutputWriterTest, DocumentBuilderTest, TestCSVResponseWriter, TestCollationKeyRangeQueries, TestStressVersions, SolrIndexConfigTest, DebugComponentTest, UpdateParamsTest, TestFastOutputStream, TestElisionMultitermQuery, TestUpdate, IndexReaderFactoryTest, TestFunctionQuery, DistributedQueryElevationComponentTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistributedQueryElevationComponentTest -Dtests.seed=F31EC422C651F5E9 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=th_TH -Dtests.timezone=Australia/LHI -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR   0.00s J1 | DistributedQueryElevationComponentTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedQueryElevationComponentTest: 
[junit4:junit4]    >    1) Thread[id=274, name=qtp898671304-274 Acceptor2 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=251, name=qtp1029168456-251 Acceptor3 SelectChannelConnector@0.0.0.0:36779, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=234, name=qtp436463023-234 Acceptor0 SelectChannelConnector@0.0.0.0:11830, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=272, name=qtp898671304-272 Acceptor0 SelectChannelConnector@0.0.0.0:53313, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    5) Thre

[...truncated too long message...]

jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=395, name=qtp1602961426-395 Acceptor1 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=374, name=qtp1219785243-374 Acceptor0 SelectChannelConnector@0.0.0.0:19621, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    5) Thread[id=345, name=qtp1954477215-345 Acceptor0 SelectChannelConnector@0.0.0.0:33948, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    6) Thread[id=376, name=qtp1219785243-376 Acceptor2 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=396, name=qtp1602961426-396 Acceptor2 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=346, name=qtp1954477215-346 Acceptor1 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    9) Thread[id=377, name=qtp1219785243-377 Acceptor3 SelectChannelConnector@0.0.0.0:19621, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   10) Thread[id=394, name=qtp1602961426-394 Acceptor0 SelectChannelConnector@0.0.0.0:55508, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=348, name=qtp1954477215-348 Acceptor3 SelectChannelConnector@0.0.0.0:33948, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=397, name=qtp1602961426-397 Acceptor3 SelectChannelConnector@0.0.0.0:55508, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([F31EC422C651F5E9]:0)
[junit4:junit4] Completed on J0 in 336.96s, 1 test, 2 errors <<< FAILURES!

[...truncated 131 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 765 ignored

Total time: 26 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 833 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/833/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:     1) Thread[id=132, name=qtp1119746574-132 Acceptor1 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=133, name=qtp1119746574-133 Acceptor2 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=84, name=qtp1696725334-84 Acceptor3 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=149, name=qtp974804600-149 Acceptor0 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=151, name=qtp974804600-151 Acceptor2 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=118, name=qtp679161401-118 Acceptor3 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=82, name=qtp1696725334-82 Acceptor1 SelectChannelConnector@0.0.0.0:38930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=150, name=qtp974804600-150 Acceptor1 SelectChannelConnector@0.0.0.0:12275, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=115, name=qtp679161401-115 Acceptor0 SelectChannelConnector@0.0.0.0:51722, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=116, name=qtp679161401-116 Acceptor1 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=131, name=qtp1119746574-131 Acceptor0 SelectChannelConnector@0.0.0.0:50985, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=83, name=qtp1696725334-83 Acceptor2 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=134, name=qtp1119746574-134 Acceptor3 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=152, name=qtp974804600-152 Acceptor3 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=81, name=qtp1696725334-81 Acceptor0 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=117, name=qtp679161401-117 Acceptor2 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest: 
   1) Thread[id=132, name=qtp1119746574-132 Acceptor1 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=133, name=qtp1119746574-133 Acceptor2 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=84, name=qtp1696725334-84 Acceptor3 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=149, name=qtp974804600-149 Acceptor0 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=151, name=qtp974804600-151 Acceptor2 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=118, name=qtp679161401-118 Acceptor3 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=82, name=qtp1696725334-82 Acceptor1 SelectChannelConnector@0.0.0.0:38930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=150, name=qtp974804600-150 Acceptor1 SelectChannelConnector@0.0.0.0:12275, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=115, name=qtp679161401-115 Acceptor0 SelectChannelConnector@0.0.0.0:51722, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=116, name=qtp679161401-116 Acceptor1 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=131, name=qtp1119746574-131 Acceptor0 SelectChannelConnector@0.0.0.0:50985, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=83, name=qtp1696725334-83 Acceptor2 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=134, name=qtp1119746574-134 Acceptor3 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=152, name=qtp974804600-152 Acceptor3 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=81, name=qtp1696725334-81 Acceptor0 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=117, name=qtp679161401-117 Acceptor2 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=132, name=qtp1119746574-132 Acceptor1 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=84, name=qtp1696725334-84 Acceptor3 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=133, name=qtp1119746574-133 Acceptor2 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=149, name=qtp974804600-149 Acceptor0 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=151, name=qtp974804600-151 Acceptor2 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=118, name=qtp679161401-118 Acceptor3 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=82, name=qtp1696725334-82 Acceptor1 SelectChannelConnector@0.0.0.0:38930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=115, name=qtp679161401-115 Acceptor0 SelectChannelConnector@0.0.0.0:51722, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=150, name=qtp974804600-150 Acceptor1 SelectChannelConnector@0.0.0.0:12275, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=116, name=qtp679161401-116 Acceptor1 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=83, name=qtp1696725334-83 Acceptor2 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=131, name=qtp1119746574-131 Acceptor0 SelectChannelConnector@0.0.0.0:50985, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=134, name=qtp1119746574-134 Acceptor3 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=152, name=qtp974804600-152 Acceptor3 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=81, name=qtp1696725334-81 Acceptor0 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=117, name=qtp679161401-117 Acceptor2 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=132, name=qtp1119746574-132 Acceptor1 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=84, name=qtp1696725334-84 Acceptor3 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=133, name=qtp1119746574-133 Acceptor2 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=149, name=qtp974804600-149 Acceptor0 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=151, name=qtp974804600-151 Acceptor2 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=118, name=qtp679161401-118 Acceptor3 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=82, name=qtp1696725334-82 Acceptor1 SelectChannelConnector@0.0.0.0:38930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=115, name=qtp679161401-115 Acceptor0 SelectChannelConnector@0.0.0.0:51722, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=150, name=qtp974804600-150 Acceptor1 SelectChannelConnector@0.0.0.0:12275, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=116, name=qtp679161401-116 Acceptor1 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=83, name=qtp1696725334-83 Acceptor2 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=131, name=qtp1119746574-131 Acceptor0 SelectChannelConnector@0.0.0.0:50985, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=134, name=qtp1119746574-134 Acceptor3 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=152, name=qtp974804600-152 Acceptor3 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=81, name=qtp1696725334-81 Acceptor0 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=117, name=qtp679161401-117 Acceptor2 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.CacheHeaderTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest:     1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest: 
   1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.CacheHeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)




Build Log:
[...truncated 7635 lines...]
[junit4:junit4] Suite: org.apache.solr.servlet.CacheHeaderTest
[junit4:junit4]   2> 0 T13 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-CacheHeaderTest-1351904893286
[junit4:junit4]   2> 2 T13 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 532 T13 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 808 T13 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:30399
[junit4:junit4]   2> 862 T13 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 863 T13 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: solr/
[junit4:junit4]   2> 863 T13 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: 'solr/'
[junit4:junit4]   2> 911 T13 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 911 T13 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 912 T13 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: solr/
[junit4:junit4]   2> 912 T13 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solr/solr.xml
[junit4:junit4]   2> 913 T13 oasc.CoreContainer.<init> New CoreContainer 1010923728
[junit4:junit4]   2> 913 T13 oasc.CoreContainer$Initializer.initialize no solr.xml file found - using default
[junit4:junit4]   2> 914 T13 oasc.CoreContainer.load Loading CoreContainer using Solr Home: 'solr/'
[junit4:junit4]   2> 914 T13 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/'
[junit4:junit4]   2> 954 T13 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1013 T13 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: solr/collection1
[junit4:junit4]   2> 1014 T13 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/collection1/'
[junit4:junit4]   2> 1139 T13 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1223 T13 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1227 T13 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1265 T13 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1821 T13 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1836 T13 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1841 T13 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1876 T13 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1886 T13 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1890 T13 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1895 T13 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1898 T13 oasc.SolrCore.<init> [collection1] Opening new SolrCore at solr/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-CacheHeaderTest-1351904893286/
[junit4:junit4]   2> 1899 T13 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79fcf790
[junit4:junit4]   2> 1900 T13 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-CacheHeaderTest-1351904893286/index/
[junit4:junit4]   2> 1901 T13 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-CacheHeaderTest-1351904893286/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1902 T13 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-CacheHeaderTest-1351904893286/index forceNew:false
[junit4:junit4]   2> 1904 T13 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@140e3010 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48423606),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1904 T13 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1905 T13 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 1907 T13 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1908 T13 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 1909 T13 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1909 T13 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 1910 T13 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1910 T13 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1911 T13 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1912 T13 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1913 T13 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1914 T13 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 1914 T13 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 1915 T13 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1916 T13 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 1916 T13 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 1917 T13 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 1918 T13 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1919 T13 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1919 T13 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1920 T13 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1921 T13 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1922 T13 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1922 T13 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1924 T13 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1924 T13 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 1925 T13 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 1930 T13 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1936 T13 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 1937 T13 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: []
[junit4:junit4]   2> 1942 T13 oass.SolrIndexSearcher.<init> Opening Searcher@1535d18b main
[junit4:junit4]   2> 1943 T13 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1944 T13 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1944 T13 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1946 T13 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 1972 T13 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2018 T13 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2018 T13 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2019 T13 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2019 T13 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2019 T13 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2020 T13 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2020 T13 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2021 T13 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2021 T13 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2022 T13 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2047 T28 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1535d18b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2053 T13 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2053 T13 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2054 T13 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2054 T13 oas.SolrJettyTestBase.createJetty Jetty Assigned Port#30399
[junit4:junit4]   2> 2062 T13 oas.SolrTestCaseJ4.setUp ###Starting testLastModified
[junit4:junit4]   2> 2085 T13 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@66e90097
[junit4:junit4]   2> 3226 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=310 
[junit4:junit4]   2> 3466 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3477 T25 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3481 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3487 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3494 T25 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3494 T13 oas.SolrTestCaseJ4.tearDown ###Ending testLastModified
[junit4:junit4]   2> 3504 T13 oas.SolrTestCaseJ4.setUp ###Starting testCacheVetoHandler
[junit4:junit4]   2> 3568 T27 C0 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@140e3010 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48423606),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3569 T27 C0 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3787 T27 C0 UPDATE [collection1] webapp=/solr path=/update params={stream.contentType=text/csv&stream.file=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/cachetest_csv1500715007tmp} {add=[100, 101, 102]} 0 271
[junit4:junit4]   2> 3789 T13 oas.SolrTestCaseJ4.tearDown ###Ending testCacheVetoHandler
[junit4:junit4]   2> 3799 T13 oas.SolrTestCaseJ4.setUp ###Starting testEtag
[junit4:junit4]   2> 3802 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3809 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=4 
[junit4:junit4]   2> 3820 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3823 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3827 T25 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3830 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3846 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3849 T27 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3850 T13 oas.SolrTestCaseJ4.tearDown ###Ending testEtag
[junit4:junit4]   2> 3858 T13 oas.SolrTestCaseJ4.setUp ###Starting testCacheControl
[junit4:junit4]   2> 3862 T25 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3866 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3873 T27 C0 REQ [collection1] webapp=/solr path=/select params={q=solr&qt=standard} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 3875 T13 oas.SolrTestCaseJ4.tearDown ###Ending testCacheControl
[junit4:junit4]   2> 3883 T13 oas.SolrTestCaseJ4.setUp ###Starting testCacheVetoException
[junit4:junit4]   2> 3888 T25 C0 oasc.SolrException.log Ignoring exception matching ignore_exception
[junit4:junit4]   2> 3888 T25 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=xyz_ignore_exception:solr} status=400 QTime=2 
[junit4:junit4]   2> 3889 T13 oas.SolrTestCaseJ4.tearDown ###Ending testCacheVetoException
[junit4:junit4]   2> 3937 T13 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1010923728
[junit4:junit4]   2> 3938 T13 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@66e90097
[junit4:junit4]   2> 3959 T13 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=3,adds=3,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 3960 T13 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 3961 T13 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 3961 T13 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 4366 T13 C0 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@140e3010 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48423606),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@140e3010 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48423606),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_TestBloomFilteredLucene41Postings_0.blm, _0_TestBloomFilteredLucene41Postings_0.tim, _0_nrm.cfs, _0_TestBloomFilteredLucene41Postings_0.tip, _0_TestBloomFilteredLucene41Postings_0.pos, _0_Direct_0.doc, _0_Pulsing41_0.doc, _0_Pulsing41_0.pos, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0_nrm.cfe, _0_TestBloomFilteredLucene41Postings_0.doc, _0_Direct_0.pos, segments_2, _0_Direct_0.tip, _0.fdx, _0.si, _0_Direct_0.tim, _0.fdt]
[junit4:junit4]   2> 4367 T13 C0 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 4368 T13 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 4638 T13 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 4738 T13 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 4739 T13 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 4750 T12 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 24846 T12 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest: 
[junit4:junit4]   2> 	   1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 24847 T12 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	   2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	   3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	   4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 27892 T12 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {timestamp=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_l=PostingsFormat(name=TestBloomFilteredLucene41Postings), id=Pulsing41(freqCutoff=15 minBlockSize=49 maxBlockSize=138), range_facet_sl=Pulsing41(freqCutoff=15 minBlockSize=49 maxBlockSize=138), range_facet_si=PostingsFormat(name=TestBloomFilteredLucene41Postings), multiDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), intDefault=PostingsFormat(name=Direct)}, sim=DefaultSimilarity, locale=sq, timezone=Atlantic/Jan_Mayen
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=477797528,total=492896256
[junit4:junit4]   2> NOTE: All tests run in this JVM: [SearchHandlerTest, CacheHeaderTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CacheHeaderTest -Dtests.seed=2E9D6974F3019263 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sq -Dtests.timezone=Atlantic/Jan_Mayen -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR   0.00s J0 | CacheHeaderTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest: 
[junit4:junit4]    >    1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=22, name=qtp948613108-22 Acceptor2 SelectChannelConnector@0.0.0.0:30399, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=21, name=qtp948613108-21 Acceptor1 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=20, name=qtp948613108-20 Acceptor0 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=23, name=qtp948613108-23 Acceptor3 SelectChannelConnector@0.0.0.0:30399, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)
[junit4:junit4] Completed on J0 in 28.13s, 5 tests, 2 errors <<< FAILURES!

[...truncated 520 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4:junit4]   2> 5 T65 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1351904922176
[junit4:junit4]   2> 7 T65 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 9 T66 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 10 T66 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 11 T66 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 108 T65 oasc.ZkTestServer.run start zk server on port:14721
[junit4:junit4]   2> 108 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@189ddde9
[junit4:junit4]   2> 109 T71 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 109 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 110 T71 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 111 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:63028
[junit4:junit4]   2> 112 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:63028
[junit4:junit4]   2> 112 T69 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 115 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460000 with negotiated timeout 10000 for client /140.211.11.196:63028
[junit4:junit4]   2> 116 T71 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460000, negotiated timeout = 10000
[junit4:junit4]   2> 116 T72 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@189ddde9 name:ZooKeeperConnection Watcher:127.0.0.1:14721 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 117 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 117 T65 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 122 T70 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac3d00a460000
[junit4:junit4]   2> 128 T72 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 129 T67 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:63028 which had sessionid 0x13ac3d00a460000
[junit4:junit4]   2> 128 T65 oaz.ZooKeeper.close Session: 0x13ac3d00a460000 closed
[junit4:junit4]   2> 130 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@27573872
[junit4:junit4]   2> 131 T73 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 131 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 132 T73 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 136 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:52262
[junit4:junit4]   2> 137 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:52262
[junit4:junit4]   2> 139 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460001 with negotiated timeout 10000 for client /140.211.11.196:52262
[junit4:junit4]   2> 139 T73 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460001, negotiated timeout = 10000
[junit4:junit4]   2> 140 T74 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27573872 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 140 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 151 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 158 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 162 T65 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 165 T65 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 169 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 189 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 203 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 204 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 221 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 222 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 226 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 227 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 232 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 233 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 237 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 238 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 242 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 243 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 247 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 248 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 252 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 253 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 257 T65 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 258 T65 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 262 T70 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ac3d00a460001
[junit4:junit4]   2> 263 T74 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 263 T65 oaz.ZooKeeper.close Session: 0x13ac3d00a460001 closed
[junit4:junit4]   2> 263 T67 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:52262 which had sessionid 0x13ac3d00a460001
[junit4:junit4]   2> 848 T65 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1116 T65 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:38930
[junit4:junit4]   2> 1157 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1158 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1158 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1191 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1192 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1192 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1192 T65 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1193 T65 oasc.CoreContainer.<init> New CoreContainer 1814681656
[junit4:junit4]   2> 1194 T65 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1194 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1229 T65 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1243 T65 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:14721/solr
[junit4:junit4]   2> 1265 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@77a477b7
[junit4:junit4]   2> 1266 T85 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 1266 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1268 T85 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 1268 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:32429
[junit4:junit4]   2> 1269 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:32429
[junit4:junit4]   2> 1272 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460002 with negotiated timeout 5000 for client /140.211.11.196:32429
[junit4:junit4]   2> 1272 T85 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460002, negotiated timeout = 5000
[junit4:junit4]   2> 1273 T86 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77a477b7 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1273 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1286 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1295 T65 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1298 T65 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38930_solr
[junit4:junit4]   2> 1302 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:delete cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:38930_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:38930_solr
[junit4:junit4]   2> 1304 T65 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38930_solr
[junit4:junit4]   2> 1308 T65 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1308 T65 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1309 T65 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1309 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1309 T65 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1310 T65 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1310 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1311 T65 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1311 T65 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1312 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1324 T65 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1335 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:delete cxid:0x15 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1342 T65 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1346 T65 oasc.Overseer.start Overseer (id=88598440980119554-127.0.0.1:38930_solr-n_0000000000) starting
[junit4:junit4]   2> 1347 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1357 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1359 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1364 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1370 T88 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1370 T65 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1376 T65 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1406 T87 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1446 T65 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1447 T65 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 1448 T65 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1449 T65 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 1451 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1452 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1453 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1587 T65 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1680 T65 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1682 T65 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1721 T65 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2249 T65 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2261 T65 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2264 T65 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2288 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2293 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2298 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2301 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2303 T65 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/control/data/
[junit4:junit4]   2> 2307 T65 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a8ab08f
[junit4:junit4]   2> 2331 T65 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/control/data/index/
[junit4:junit4]   2> 2332 T65 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2333 T65 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/control/data/index forceNew:false
[junit4:junit4]   2> 2342 T65 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@762589c3; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2343 T65 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2344 T65 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 2346 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2347 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 2347 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2349 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 2349 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2350 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2350 T65 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2351 T65 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2352 T65 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2353 T65 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 2353 T65 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 2354 T65 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2355 T65 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2355 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2356 T65 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2357 T65 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2357 T65 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2358 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2359 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2359 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2360 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2361 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2362 T65 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2363 T65 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 2363 T65 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 2368 T65 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2372 T65 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 2374 T65 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 2378 T65 oass.SolrIndexSearcher.<init> Opening Searcher@201ba640 main
[junit4:junit4]   2> 2396 T65 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2396 T65 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2397 T65 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 2415 T65 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2470 T93 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@201ba640 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2478 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2479 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x56 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2483 T65 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2484 T65 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:38930/solr shardId:control_shard
[junit4:junit4]   2> 2495 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 2509 T65 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/control_shard/election
[junit4:junit4]   2> 2529 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:delete cxid:0x65 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2530 T65 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2531 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x66 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2915 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2917 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:38930_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:38930/solr"}
[junit4:junit4]   2> 2926 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x6d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2954 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3035 T65 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 3036 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 3037 T65 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38930/solr/collection1/
[junit4:junit4]   2> 3038 T65 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 3038 T65 oasc.SyncStrategy.syncToMe http://127.0.0.1:38930/solr/collection1/ has no replicas
[junit4:junit4]   2> 3039 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38930/solr/collection1/
[junit4:junit4]   2> 3039 T65 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/control_shard
[junit4:junit4]   2> 3047 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460002 type:create cxid:0x8b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3465 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3514 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3567 T65 oasc.ZkController.register We are http://127.0.0.1:38930/solr/collection1/ and leader is http://127.0.0.1:38930/solr/collection1/
[junit4:junit4]   2> 3571 T65 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38930/solr
[junit4:junit4]   2> 3572 T65 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3572 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3575 T65 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3577 T65 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3577 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3647 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3833 T65 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 3836 T65 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:13518
[junit4:junit4]   2> 3837 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3838 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841
[junit4:junit4]   2> 3838 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/'
[junit4:junit4]   2> 3875 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 3875 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3876 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841
[junit4:junit4]   2> 3877 T65 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/solr.xml
[junit4:junit4]   2> 3877 T65 oasc.CoreContainer.<init> New CoreContainer 428806951
[junit4:junit4]   2> 3878 T65 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/'
[junit4:junit4]   2> 3878 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/'
[junit4:junit4]   2> 3906 T65 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3924 T65 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:14721/solr
[junit4:junit4]   2> 3925 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@6ffe8714
[junit4:junit4]   2> 3926 T103 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 3926 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3928 T103 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 3928 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:44255
[junit4:junit4]   2> 3929 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:44255
[junit4:junit4]   2> 4089 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460003 with negotiated timeout 5000 for client /140.211.11.196:44255
[junit4:junit4]   2> 4089 T103 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460003, negotiated timeout = 5000
[junit4:junit4]   2> 4090 T104 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ffe8714 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4090 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4091 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4094 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4104 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4104 T65 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13518_solr
[junit4:junit4]   2> 4105 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:38930_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:38930/solr"}
[junit4:junit4]   2> 4105 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13518_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13518_solr
[junit4:junit4]   2> 4107 T65 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13518_solr
[junit4:junit4]   2> 4120 T65 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 4120 T65 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4121 T65 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 4121 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4122 T65 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4122 T65 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4123 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4124 T65 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4124 T65 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4125 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 4143 T86 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4144 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 4151 T65 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 4159 T65 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/collection1
[junit4:junit4]   2> 4160 T65 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 4161 T65 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 4162 T65 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 4164 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/collection1/'
[junit4:junit4]   2> 4166 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty1-1351904925841/collection1/lib/README' to classloader
[junit4:junit4]   2> 4167 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty1-1351904925841/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 4234 T65 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 4313 T65 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 4315 T65 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 4345 T65 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 5006 T65 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5017 T65 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5020 T65 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5044 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5049 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5053 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5055 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5058 T65 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351904925841/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty1/
[junit4:junit4]   2> 5059 T65 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a8ab08f
[junit4:junit4]   2> 5060 T65 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty1/index/
[junit4:junit4]   2> 5060 T65 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5061 T65 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty1/index forceNew:false
[junit4:junit4]   2> 5067 T65 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5abe753a; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5068 T65 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5068 T65 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 5071 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5071 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 5072 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 5072 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 5074 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 5074 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 5075 T65 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 5075 T65 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5076 T65 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5077 T65 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 5078 T65 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 5078 T65 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5079 T65 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 5080 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 5080 T65 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 5081 T65 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5082 T65 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5083 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5083 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5084 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5085 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5086 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5086 T65 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5087 T65 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 5088 T65 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 5094 T65 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5100 T65 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 5102 T65 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 5106 T65 oass.SolrIndexSearcher.<init> Opening Searcher@22a79c31 main
[junit4:junit4]   2> 5107 T65 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5107 T65 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5108 T65 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 5114 T65 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 5174 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22a79c31 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5185 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 5650 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5652 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:13518_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:13518/solr"}
[junit4:junit4]   2> 5652 T87 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 5743 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5743 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6189 T65 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 6190 T65 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:13518/solr shardId:shard1
[junit4:junit4]   2> 6191 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 6214 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:delete cxid:0x42 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 6215 T65 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 6216 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:create cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6225 T65 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 6226 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 6227 T65 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:13518/solr/collection1/
[junit4:junit4]   2> 6227 T65 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 6228 T65 oasc.SyncStrategy.syncToMe http://127.0.0.1:13518/solr/collection1/ has no replicas
[junit4:junit4]   2> 6228 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:13518/solr/collection1/
[junit4:junit4]   2> 6229 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 6235 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460003 type:create cxid:0x4d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6249 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6267 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6267 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6295 T65 oasc.ZkController.register We are http://127.0.0.1:13518/solr/collection1/ and leader is http://127.0.0.1:13518/solr/collection1/
[junit4:junit4]   2> 6295 T65 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13518/solr
[junit4:junit4]   2> 6296 T65 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 6296 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6299 T65 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6300 T65 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 6301 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 6302 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 6472 T65 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 6475 T65 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:51722
[junit4:junit4]   2> 6477 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6477 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487
[junit4:junit4]   2> 6478 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/'
[junit4:junit4]   2> 6523 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 6523 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6524 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487
[junit4:junit4]   2> 6525 T65 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/solr.xml
[junit4:junit4]   2> 6525 T65 oasc.CoreContainer.<init> New CoreContainer 845986605
[junit4:junit4]   2> 6526 T65 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/'
[junit4:junit4]   2> 6527 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/'
[junit4:junit4]   2> 6553 T65 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 6570 T65 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:14721/solr
[junit4:junit4]   2> 6571 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@690463c3
[junit4:junit4]   2> 6572 T119 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 6572 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6573 T119 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 6574 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:43782
[junit4:junit4]   2> 6574 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:43782
[junit4:junit4]   2> 6577 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460004 with negotiated timeout 5000 for client /140.211.11.196:43782
[junit4:junit4]   2> 6577 T119 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460004, negotiated timeout = 5000
[junit4:junit4]   2> 6578 T120 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@690463c3 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6578 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6579 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6594 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6596 T65 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51722_solr
[junit4:junit4]   2> 6597 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:51722_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:51722_solr
[junit4:junit4]   2> 6603 T65 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51722_solr
[junit4:junit4]   2> 6607 T86 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 6607 T104 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 6608 T65 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 6608 T65 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 6609 T65 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 6610 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 6610 T65 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 6611 T65 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 6611 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 6612 T65 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 6612 T65 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 6613 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 6632 T65 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 6640 T65 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/collection1
[junit4:junit4]   2> 6641 T65 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 6642 T65 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 6642 T65 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 6644 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/collection1/'
[junit4:junit4]   2> 6646 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty2-1351904928487/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 6647 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty2-1351904928487/collection1/lib/README' to classloader
[junit4:junit4]   2> 6709 T65 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 6776 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6777 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:13518_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:13518/solr"}
[junit4:junit4]   2> 6789 T65 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 6791 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6791 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6791 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6791 T65 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 6814 T65 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 7498 T65 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 7509 T65 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 7512 T65 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 7533 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7538 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7542 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7544 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7546 T65 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351904928487/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty2/
[junit4:junit4]   2> 7547 T65 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a8ab08f
[junit4:junit4]   2> 7548 T65 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty2/index/
[junit4:junit4]   2> 7549 T65 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 7549 T65 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty2/index forceNew:false
[junit4:junit4]   2> 7559 T65 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@71c0d0a8; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 7559 T65 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 7560 T65 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 7562 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 7563 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 7564 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 7564 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 7565 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 7565 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 7566 T65 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 7566 T65 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 7567 T65 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 7568 T65 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 7569 T65 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 7569 T65 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 7570 T65 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 7571 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 7571 T65 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 7572 T65 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 7573 T65 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7573 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7574 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7575 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7576 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7576 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7577 T65 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7578 T65 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 7579 T65 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 7584 T65 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 7587 T65 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 7589 T65 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 7592 T65 oass.SolrIndexSearcher.<init> Opening Searcher@67219b8c main
[junit4:junit4]   2> 7593 T65 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 7594 T65 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7594 T65 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 7600 T65 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 7650 T125 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67219b8c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 7658 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7801 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7802 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:51722_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:51722/solr"}
[junit4:junit4]   2> 7823 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7823 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7823 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8161 T65 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 8161 T65 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:51722/solr shardId:shard2
[junit4:junit4]   2> 8163 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 8179 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:delete cxid:0x42 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 8180 T65 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 8181 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:create cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8191 T65 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 8192 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 8193 T65 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51722/solr/collection1/
[junit4:junit4]   2> 8193 T65 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 8194 T65 oasc.SyncStrategy.syncToMe http://127.0.0.1:51722/solr/collection1/ has no replicas
[junit4:junit4]   2> 8194 T65 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51722/solr/collection1/
[junit4:junit4]   2> 8195 T65 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 8199 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460004 type:create cxid:0x4c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8329 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8347 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8347 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8347 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8355 T65 oasc.ZkController.register We are http://127.0.0.1:51722/solr/collection1/ and leader is http://127.0.0.1:51722/solr/collection1/
[junit4:junit4]   2> 8355 T65 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51722/solr
[junit4:junit4]   2> 8355 T65 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8356 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8358 T65 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8359 T65 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 8359 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8360 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8483 T65 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 8487 T65 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:50985
[junit4:junit4]   2> 8488 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8488 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540
[junit4:junit4]   2> 8489 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/'
[junit4:junit4]   2> 8525 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8526 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8526 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540
[junit4:junit4]   2> 8527 T65 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/solr.xml
[junit4:junit4]   2> 8528 T65 oasc.CoreContainer.<init> New CoreContainer 1463607684
[junit4:junit4]   2> 8528 T65 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/'
[junit4:junit4]   2> 8529 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/'
[junit4:junit4]   2> 8555 T65 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8572 T65 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:14721/solr
[junit4:junit4]   2> 8573 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@3b39d369
[junit4:junit4]   2> 8574 T135 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 8574 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8575 T135 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 8576 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:27224
[junit4:junit4]   2> 8576 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:27224
[junit4:junit4]   2> 8579 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460005 with negotiated timeout 5000 for client /140.211.11.196:27224
[junit4:junit4]   2> 8579 T135 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460005, negotiated timeout = 5000
[junit4:junit4]   2> 8579 T136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b39d369 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8580 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8581 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460005 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8583 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460005 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8593 T65 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50985_solr
[junit4:junit4]   2> 8594 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:50985_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:50985_solr
[junit4:junit4]   2> 8595 T65 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50985_solr
[junit4:junit4]   2> 8599 T104 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 8600 T86 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 8600 T120 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 8600 T65 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 8601 T65 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8601 T65 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 8602 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8602 T65 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8603 T65 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8603 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8604 T65 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8605 T65 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8605 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 8625 T65 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 8633 T65 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/collection1
[junit4:junit4]   2> 8633 T65 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 8634 T65 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 8635 T65 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 8637 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/collection1/'
[junit4:junit4]   2> 8638 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty3-1351904930540/collection1/lib/README' to classloader
[junit4:junit4]   2> 8639 T65 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty3-1351904930540/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 8700 T65 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 8776 T65 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 8778 T65 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 8801 T65 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 8862 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8864 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:51722_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:51722/solr"}
[junit4:junit4]   2> 8871 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 8871 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 8871 T136 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 8871 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9461 T65 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 9471 T65 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 9474 T65 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 9494 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9498 T65 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9502 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9504 T65 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9506 T65 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351904930540/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty3/
[junit4:junit4]   2> 9507 T65 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2a8ab08f
[junit4:junit4]   2> 9508 T65 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty3/index/
[junit4:junit4]   2> 9509 T65 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 9510 T65 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty3/index forceNew:false
[junit4:junit4]   2> 9515 T65 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351904922176/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@4c5e43ee; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9516 T65 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 9517 T65 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 9520 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 9520 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 9521 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 9521 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 9522 T65 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 9522 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 9523 T65 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 9524 T65 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 9524 T65 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 9525 T65 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 9526 T65 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 9527 T65 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 9527 T65 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 9528 T65 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 9528 T65 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 9529 T65 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 9530 T65 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9530 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9531 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9532 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9532 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9533 T65 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9534 T65 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9534 T65 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 9535 T65 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 9540 T65 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 9543 T65 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 9545 T65 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 9548 T65 oass.SolrIndexSearcher.<init> Opening Searcher@72cd429b main
[junit4:junit4]   2> 9549 T65 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 9549 T65 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 9550 T65 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 9556 T65 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 9610 T141 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@72cd429b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 9617 T65 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 9885 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9886 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50985_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50985/solr"}
[junit4:junit4]   2> 9900 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9900 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9900 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9900 T136 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10120 T65 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 10120 T65 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:50985/solr shardId:shard1
[junit4:junit4]   2> 10125 T65 oasc.ZkController.register We are http://127.0.0.1:50985/solr/collection1/ and leader is http://127.0.0.1:13518/solr/collection1/
[junit4:junit4]   2> 10125 T65 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50985/solr
[junit4:junit4]   2> 10126 T65 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 10127 T65 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@58a40787 url=http://127.0.0.1:50985/solr/collection1 node=127.0.0.1:50985_solr
[junit4:junit4]   2> 10134 T142 C2 P50985 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 10134 T65 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10134 T142 C2 P50985 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 10135 T142 C2 P50985 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 10136 T65 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 10137 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 10137 T65 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 10137 T142 C2 P50985 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:13518/solr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 10145 T142 C2 P50985 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 10163 T142 C2 P50985 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50985/solr START replicas=[http://127.0.0.1:13518/solr/collection1/] nUpdates=100
[junit4:junit4]   2> 10177 T142 C2 P50985 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 10178 T142 C2 P50985 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 10179 T142 C2 P50985 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 10179 T142 C2 P50985 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 10337 T65 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 10341 T65 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:12275
[junit4:junit4]   2> 10342 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 10342 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326
[junit4:junit4]   2> 10343 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326/'
[junit4:junit4]   2> 10406 T87 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10407 T87 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50985_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50985/solr"}
[junit4:junit4]   2> 10423 T104 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10423 T120 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10423 T136 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10423 T86 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10517 T65 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 10518 T65 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 10519 T65 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326
[junit4:junit4]   2> 10519 T65 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326/solr.xml
[junit4:junit4]   2> 10520 T65 oasc.CoreContainer.<init> New CoreContainer 1292438614
[junit4:junit4]   2> 10521 T65 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326/'
[junit4:junit4]   2> 10522 T65 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351904932326/'
[junit4:junit4]   2> 10869 T65 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 10984 T65 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:14721/solr
[junit4:junit4]   2> 10985 T65 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:14721/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@6d1cf4e5
[junit4:junit4]   2> 10986 T156 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:14721
[junit4:junit4]   2> 10986 T65 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 10987 T156 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:14721, initiating session
[junit4:junit4]   2> 10988 T67 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:63117
[junit4:junit4]   2> 10988 T67 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:63117
[junit4:junit4]   2> 10991 T69 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ac3d00a460006 with negotiated timeout 5000 for client /140.211.11.196:63117
[junit4:junit4]   2> 10991 T156 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:14721, sessionid = 0x13ac3d00a460006, negotiated timeout = 5000
[junit4:junit4]   2> 10991 T157 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d1cf4e5 name:ZooKeeperConnection Watcher:127.0.0.1:14721/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 10992 T65 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 10993 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460006 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 10995 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460006 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11006 T65 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:12275_solr
[junit4:junit4]   2> 11007 T70 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ac3d00a460006 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:12275_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:12275_solr
[junit4:junit4]   2> 11008 T65 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:12275_solr
[junit4:junit4]   2> 11012 T104 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11012 T120 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11012 T86 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11012 T136 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 11013 T65 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 11014 T65 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 11014 T65 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 11015 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 11016 T65 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 11017 T65 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 11017 T65 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 11018 T65 oashc.HttpShardHandle

[...truncated too long message...]

dThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=82, name=qtp1696725334-82 Acceptor1 SelectChannelConnector@0.0.0.0:38930, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=115, name=qtp679161401-115 Acceptor0 SelectChannelConnector@0.0.0.0:51722, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    9) Thread[id=150, name=qtp974804600-150 Acceptor1 SelectChannelConnector@0.0.0.0:12275, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   10) Thread[id=116, name=qtp679161401-116 Acceptor1 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=83, name=qtp1696725334-83 Acceptor2 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=131, name=qtp1119746574-131 Acceptor0 SelectChannelConnector@0.0.0.0:50985, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=134, name=qtp1119746574-134 Acceptor3 SelectChannelConnector@0.0.0.0:50985, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=152, name=qtp974804600-152 Acceptor3 SelectChannelConnector@0.0.0.0:12275, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=81, name=qtp1696725334-81 Acceptor0 SelectChannelConnector@0.0.0.0:38930, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=117, name=qtp679161401-117 Acceptor2 SelectChannelConnector@0.0.0.0:51722, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E9D6974F3019263]:0)
[junit4:junit4] Completed on J1 in 75.40s, 1 test, 2 errors <<< FAILURES!

[...truncated 170 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 861 ignored

Total time: 19 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 832 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/832/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.update.PeerSyncTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.update.PeerSyncTest:     1) Thread[id=127, name=qtp1806030550-127 Acceptor0 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=104, name=qtp1585929710-104 Acceptor3 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=90, name=qtp1025601370-90 Acceptor3 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=117, name=qtp221827836-117 Acceptor2 SelectChannelConnector@0.0.0.0:39858, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=130, name=qtp1806030550-130 Acceptor3 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=115, name=qtp221827836-115 Acceptor0 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=118, name=qtp221827836-118 Acceptor3 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=116, name=qtp221827836-116 Acceptor1 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=101, name=qtp1585929710-101 Acceptor0 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=102, name=qtp1585929710-102 Acceptor1 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=103, name=qtp1585929710-103 Acceptor2 SelectChannelConnector@0.0.0.0:39857, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=88, name=qtp1025601370-88 Acceptor1 SelectChannelConnector@0.0.0.0:39856, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=87, name=qtp1025601370-87 Acceptor0 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=89, name=qtp1025601370-89 Acceptor2 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=129, name=qtp1806030550-129 Acceptor2 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=128, name=qtp1806030550-128 Acceptor1 SelectChannelConnector@0.0.0.0:39859, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.update.PeerSyncTest: 
   1) Thread[id=127, name=qtp1806030550-127 Acceptor0 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=104, name=qtp1585929710-104 Acceptor3 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=90, name=qtp1025601370-90 Acceptor3 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=117, name=qtp221827836-117 Acceptor2 SelectChannelConnector@0.0.0.0:39858, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=130, name=qtp1806030550-130 Acceptor3 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=115, name=qtp221827836-115 Acceptor0 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=118, name=qtp221827836-118 Acceptor3 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=116, name=qtp221827836-116 Acceptor1 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=101, name=qtp1585929710-101 Acceptor0 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=102, name=qtp1585929710-102 Acceptor1 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=103, name=qtp1585929710-103 Acceptor2 SelectChannelConnector@0.0.0.0:39857, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=88, name=qtp1025601370-88 Acceptor1 SelectChannelConnector@0.0.0.0:39856, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=87, name=qtp1025601370-87 Acceptor0 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=89, name=qtp1025601370-89 Acceptor2 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=129, name=qtp1806030550-129 Acceptor2 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=128, name=qtp1806030550-128 Acceptor1 SelectChannelConnector@0.0.0.0:39859, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([A461CFFBFF73C82E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.PeerSyncTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=104, name=qtp1585929710-104 Acceptor3 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=127, name=qtp1806030550-127 Acceptor0 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=90, name=qtp1025601370-90 Acceptor3 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=117, name=qtp221827836-117 Acceptor2 SelectChannelConnector@0.0.0.0:39858, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=115, name=qtp221827836-115 Acceptor0 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=130, name=qtp1806030550-130 Acceptor3 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=101, name=qtp1585929710-101 Acceptor0 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=116, name=qtp221827836-116 Acceptor1 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=118, name=qtp221827836-118 Acceptor3 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=102, name=qtp1585929710-102 Acceptor1 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=88, name=qtp1025601370-88 Acceptor1 SelectChannelConnector@0.0.0.0:39856, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=103, name=qtp1585929710-103 Acceptor2 SelectChannelConnector@0.0.0.0:39857, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=87, name=qtp1025601370-87 Acceptor0 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=89, name=qtp1025601370-89 Acceptor2 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=129, name=qtp1806030550-129 Acceptor2 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=128, name=qtp1806030550-128 Acceptor1 SelectChannelConnector@0.0.0.0:39859, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=104, name=qtp1585929710-104 Acceptor3 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=127, name=qtp1806030550-127 Acceptor0 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=90, name=qtp1025601370-90 Acceptor3 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=117, name=qtp221827836-117 Acceptor2 SelectChannelConnector@0.0.0.0:39858, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=115, name=qtp221827836-115 Acceptor0 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=130, name=qtp1806030550-130 Acceptor3 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=101, name=qtp1585929710-101 Acceptor0 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=116, name=qtp221827836-116 Acceptor1 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=118, name=qtp221827836-118 Acceptor3 SelectChannelConnector@0.0.0.0:39858, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=102, name=qtp1585929710-102 Acceptor1 SelectChannelConnector@0.0.0.0:39857, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=88, name=qtp1025601370-88 Acceptor1 SelectChannelConnector@0.0.0.0:39856, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=103, name=qtp1585929710-103 Acceptor2 SelectChannelConnector@0.0.0.0:39857, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=87, name=qtp1025601370-87 Acceptor0 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=89, name=qtp1025601370-89 Acceptor2 SelectChannelConnector@0.0.0.0:39856, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=129, name=qtp1806030550-129 Acceptor2 SelectChannelConnector@0.0.0.0:39859, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=128, name=qtp1806030550-128 Acceptor1 SelectChannelConnector@0.0.0.0:39859, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([A461CFFBFF73C82E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test:     1) Thread[id=224, name=qtp1229973302-224 Acceptor3 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=371, name=qtp1711596206-371 Acceptor2 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=240, name=qtp2115930648-240 Acceptor1 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=223, name=qtp1229973302-223 Acceptor2 SelectChannelConnector@0.0.0.0:39123, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=174, name=qtp1257641869-174 Acceptor1 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=241, name=qtp2115930648-241 Acceptor2 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=239, name=qtp2115930648-239 Acceptor0 SelectChannelConnector@0.0.0.0:65310, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=369, name=qtp1711596206-369 Acceptor0 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=176, name=qtp1257641869-176 Acceptor3 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=370, name=qtp1711596206-370 Acceptor1 SelectChannelConnector@0.0.0.0:22970, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=222, name=qtp1229973302-222 Acceptor1 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=221, name=qtp1229973302-221 Acceptor0 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=372, name=qtp1711596206-372 Acceptor3 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=242, name=qtp2115930648-242 Acceptor3 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=173, name=qtp1257641869-173 Acceptor0 SelectChannelConnector@0.0.0.0:39867, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=175, name=qtp1257641869-175 Acceptor2 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test: 
   1) Thread[id=224, name=qtp1229973302-224 Acceptor3 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=371, name=qtp1711596206-371 Acceptor2 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=240, name=qtp2115930648-240 Acceptor1 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=223, name=qtp1229973302-223 Acceptor2 SelectChannelConnector@0.0.0.0:39123, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=174, name=qtp1257641869-174 Acceptor1 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=241, name=qtp2115930648-241 Acceptor2 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=239, name=qtp2115930648-239 Acceptor0 SelectChannelConnector@0.0.0.0:65310, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=369, name=qtp1711596206-369 Acceptor0 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=176, name=qtp1257641869-176 Acceptor3 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=370, name=qtp1711596206-370 Acceptor1 SelectChannelConnector@0.0.0.0:22970, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=222, name=qtp1229973302-222 Acceptor1 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=221, name=qtp1229973302-221 Acceptor0 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=372, name=qtp1711596206-372 Acceptor3 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=242, name=qtp2115930648-242 Acceptor3 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=173, name=qtp1257641869-173 Acceptor0 SelectChannelConnector@0.0.0.0:39867, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=175, name=qtp1257641869-175 Acceptor2 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([A461CFFBFF73C82E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=224, name=qtp1229973302-224 Acceptor3 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=240, name=qtp2115930648-240 Acceptor1 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=371, name=qtp1711596206-371 Acceptor2 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=223, name=qtp1229973302-223 Acceptor2 SelectChannelConnector@0.0.0.0:39123, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=174, name=qtp1257641869-174 Acceptor1 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=239, name=qtp2115930648-239 Acceptor0 SelectChannelConnector@0.0.0.0:65310, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=241, name=qtp2115930648-241 Acceptor2 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=176, name=qtp1257641869-176 Acceptor3 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=369, name=qtp1711596206-369 Acceptor0 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=370, name=qtp1711596206-370 Acceptor1 SelectChannelConnector@0.0.0.0:22970, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=221, name=qtp1229973302-221 Acceptor0 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=222, name=qtp1229973302-222 Acceptor1 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=372, name=qtp1711596206-372 Acceptor3 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=242, name=qtp2115930648-242 Acceptor3 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=173, name=qtp1257641869-173 Acceptor0 SelectChannelConnector@0.0.0.0:39867, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=175, name=qtp1257641869-175 Acceptor2 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=224, name=qtp1229973302-224 Acceptor3 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=240, name=qtp2115930648-240 Acceptor1 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=371, name=qtp1711596206-371 Acceptor2 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=223, name=qtp1229973302-223 Acceptor2 SelectChannelConnector@0.0.0.0:39123, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=174, name=qtp1257641869-174 Acceptor1 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=239, name=qtp2115930648-239 Acceptor0 SelectChannelConnector@0.0.0.0:65310, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=241, name=qtp2115930648-241 Acceptor2 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=176, name=qtp1257641869-176 Acceptor3 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=369, name=qtp1711596206-369 Acceptor0 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=370, name=qtp1711596206-370 Acceptor1 SelectChannelConnector@0.0.0.0:22970, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=221, name=qtp1229973302-221 Acceptor0 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=222, name=qtp1229973302-222 Acceptor1 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=372, name=qtp1711596206-372 Acceptor3 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=242, name=qtp2115930648-242 Acceptor3 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=173, name=qtp1257641869-173 Acceptor0 SelectChannelConnector@0.0.0.0:39867, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=175, name=qtp1257641869-175 Acceptor2 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([A461CFFBFF73C82E]:0)




Build Log:
[...truncated 7738 lines...]
[junit4:junit4] Suite: org.apache.solr.update.PeerSyncTest
[junit4:junit4]   2> 5 T80 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 511 T80 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 815 T80 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:39856
[junit4:junit4]   2> 843 T80 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 844 T80 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 844 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 875 T80 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 876 T80 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 876 T80 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 877 T80 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 877 T80 oasc.CoreContainer.<init> New CoreContainer 940667589
[junit4:junit4]   2> 878 T80 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 878 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 898 T80 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 916 T80 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 917 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 918 T80 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 918 T80 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 954 T80 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 996 T80 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 997 T80 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1020 T80 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1530 T80 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1542 T80 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1546 T80 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1561 T80 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1565 T80 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1569 T80 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1571 T80 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1573 T80 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351878403370/control/data/
[junit4:junit4]   2> 1574 T80 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 1575 T80 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351878403370/control/data/index/
[junit4:junit4]   2> 1575 T80 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351878403370/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1576 T80 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351878403370/control/data/index forceNew:false
[junit4:junit4]   2> 1579 T80 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@502a3135 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3b7c680),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1580 T80 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1581 T80 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1581 T80 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1582 T80 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1582 T80 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1588 T80 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1593 T80 oass.SolrIndexSearcher.<init> Opening Searcher@3d2da4a6 main
[junit4:junit4]   2> 1593 T80 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1594 T80 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1595 T80 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1595 T80 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1596 T80 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1596 T80 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1596 T80 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1597 T80 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1597 T80 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1598 T80 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1598 T80 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1598 T80 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1599 T80 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1611 T95 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3d2da4a6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1611 T80 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1612 T80 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 1612 T80 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1664 T80 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1682 T80 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1684 T80 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:39857
[junit4:junit4]   2> 1685 T80 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1685 T80 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1686 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1717 T80 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1717 T80 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1718 T80 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1718 T80 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1719 T80 oasc.CoreContainer.<init> New CoreContainer 187247026
[junit4:junit4]   2> 1719 T80 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1720 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1739 T80 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1756 T80 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1756 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1758 T80 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1758 T80 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1795 T80 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1837 T80 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 1838 T80 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1855 T80 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2339 T80 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2350 T80 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2353 T80 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2368 T80 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2372 T80 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2376 T80 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2377 T80 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2379 T80 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351878403370/shard0/data/
[junit4:junit4]   2> 2379 T80 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 2380 T80 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351878403370/shard0/data/index/
[junit4:junit4]   2> 2381 T80 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351878403370/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2382 T80 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351878403370/shard0/data/index forceNew:false
[junit4:junit4]   2> 2385 T80 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2385 T80 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2386 T80 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2387 T80 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2387 T80 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2388 T80 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2393 T80 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2398 T80 oass.SolrIndexSearcher.<init> Opening Searcher@4302a01f main
[junit4:junit4]   2> 2399 T80 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 2400 T80 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2401 T80 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2401 T80 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2402 T80 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2402 T80 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2403 T80 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2403 T80 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2403 T80 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2404 T80 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2404 T80 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2405 T80 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2405 T80 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2416 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4302a01f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2416 T80 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2417 T80 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2417 T80 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2418 T80 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2426 T80 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2428 T80 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:39858
[junit4:junit4]   2> 2429 T80 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2430 T80 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2430 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2455 T80 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2455 T80 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2456 T80 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2456 T80 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2456 T80 oasc.CoreContainer.<init> New CoreContainer 1323538699
[junit4:junit4]   2> 2457 T80 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2457 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2477 T80 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2493 T80 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2494 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2495 T80 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2496 T80 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2528 T80 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2567 T80 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 2568 T80 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2582 T80 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 3075 T80 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 3086 T80 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 3089 T80 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 3103 T80 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3107 T80 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3110 T80 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3111 T80 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3113 T80 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351878403370/shard1/data/
[junit4:junit4]   2> 3113 T80 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 3114 T80 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351878403370/shard1/data/index/
[junit4:junit4]   2> 3115 T80 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351878403370/shard1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 3116 T80 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351878403370/shard1/data/index forceNew:false
[junit4:junit4]   2> 3119 T80 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3119 T80 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3120 T80 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 3121 T80 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 3121 T80 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 3122 T80 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 3127 T80 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 3182 T80 oass.SolrIndexSearcher.<init> Opening Searcher@6fe99db4 main
[junit4:junit4]   2> 3182 T80 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3183 T80 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 3184 T80 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 3184 T80 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 3185 T80 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3185 T80 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 3185 T80 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3186 T80 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3186 T80 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3186 T80 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3187 T80 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3187 T80 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3188 T80 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 3208 T121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6fe99db4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 3208 T80 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 3209 T80 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3209 T80 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3210 T80 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3218 T80 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 3221 T80 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:39859
[junit4:junit4]   2> 3222 T80 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3222 T80 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 3222 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3244 T80 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 3245 T80 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3245 T80 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 3246 T80 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 3246 T80 oasc.CoreContainer.<init> New CoreContainer 552296573
[junit4:junit4]   2> 3247 T80 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3247 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3264 T80 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3279 T80 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 3280 T80 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 3281 T80 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 3281 T80 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 3313 T80 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 3351 T80 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 3352 T80 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 3373 T80 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 3928 T80 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 3938 T80 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 3941 T80 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 3956 T80 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3962 T80 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3967 T80 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3969 T80 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3971 T80 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351878403370/shard2/data/
[junit4:junit4]   2> 3971 T80 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 3973 T80 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351878403370/shard2/data/index/
[junit4:junit4]   2> 3974 T80 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351878403370/shard2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 3975 T80 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351878403370/shard2/data/index forceNew:false
[junit4:junit4]   2> 3980 T80 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4ffe0816 lockFactory=org.apache.lucene.store.NativeFSLockFactory@604c9c17; maxCacheMB=0.701171875 maxMergeSizeMB=0.5654296875)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3981 T80 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3982 T80 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 3983 T80 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 3984 T80 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 3984 T80 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 3992 T80 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 3999 T80 oass.SolrIndexSearcher.<init> Opening Searcher@4631c43f main
[junit4:junit4]   2> 3999 T80 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4000 T80 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 4000 T80 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 4001 T80 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 4001 T80 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4002 T80 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 4002 T80 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4002 T80 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4003 T80 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4003 T80 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4004 T80 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4004 T80 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4004 T80 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 4016 T136 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4631c43f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 4016 T80 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 4017 T80 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 4017 T80 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 4017 T80 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@6dc8f3cd
[junit4:junit4]   2> 5092 T108 C3 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5093 T108 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5222 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=165 
[junit4:junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@23f9e6e5
[junit4:junit4]   2> 5418 T131 C4 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 5435 T131 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:39857/solr] nUpdates=100
[junit4:junit4]   2> 5443 T131 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 5444 T131 C4 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:39857/solr&version=2&getVersions=100} status=0 QTime=77 
[junit4:junit4]   2> 5453 T120 C4 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5454 T120 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5464 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=14 
[junit4:junit4]   2> 5468 T131 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:39857/solr] nUpdates=100
[junit4:junit4]   2> 5477 T107 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=6 
[junit4:junit4]   2> 5478 T106 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=11 
[junit4:junit4]   2> 5479 T131 C4 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 1 versions from 127.0.0.1:39857/solr
[junit4:junit4]   2> 5479 T131 C4 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Our versions are newer. ourLowThreshold=1 otherHigh=1
[junit4:junit4]   2> 5479 T131 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 5480 T131 C4 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:39857/solr&version=2&getVersions=100} status=0 QTime=12 
[junit4:junit4]   2> 5486 T108 C3 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5767 T108 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_2,generation=2,filenames=[_0_Asserting_0.tim, _0.fnm, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _0_Lucene41_0.pos, _0_nrm.cfs, _0_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _0_Lucene41_0.doc, _0_SimpleText_0.pst, _0_nrm.cfe, _0_MockVariableIntBlock_0.frq, _0_MockVariableIntBlock_0.pos, segments_2, _0.fdx, _0.si, _0_Asserting_0.pos, _0_Lucene41_0.tim, _0_MockVariableIntBlock_0.tii, _0.fdt, _0_Lucene41_0.tip, _0_Asserting_0.doc]
[junit4:junit4]   2> 5768 T108 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 5990 T108 C3 oass.SolrIndexSearcher.<init> Opening Searcher@4c76ec2d main
[junit4:junit4]   2> 5990 T108 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5992 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c76ec2d main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 5992 T108 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5993 T108 C3 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 507
[junit4:junit4]   2> 5997 T120 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6003 T120 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_2,generation=2,filenames=[_0_Asserting_0.tim, _0.fnm, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _0_Lucene41_0.pos, _0_nrm.cfs, _0_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _0_Lucene41_0.doc, _0_SimpleText_0.pst, _0_nrm.cfe, _0_MockVariableIntBlock_0.frq, _0_MockVariableIntBlock_0.pos, segments_2, _0.fdx, _0.si, _0_Asserting_0.pos, _0_Lucene41_0.tim, _0_MockVariableIntBlock_0.tii, _0.fdt, _0_Lucene41_0.tip, _0_Asserting_0.doc]
[junit4:junit4]   2> 6004 T120 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 6010 T120 C4 oass.SolrIndexSearcher.<init> Opening Searcher@124ef122 main
[junit4:junit4]   2> 6011 T120 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6012 T121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@124ef122 main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 6012 T120 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6012 T120 C4 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 15
[junit4:junit4]   2> 6039 T106 C3 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 6072 T131 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 6131 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=7 
[junit4:junit4]   2> 6136 T120 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:39857/solr] nUpdates=100
[junit4:junit4]   2> 6140 T106 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=0 
[junit4:junit4]   2> 6141 T120 C4 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 2 versions from 127.0.0.1:39857/solr
[junit4:junit4]   2> 6142 T120 C4 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:39857/solrn=1 versions=[2]
[junit4:junit4]   2> 6146 T107 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=2&qt=/get&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 6155 T120 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 6156 T120 C4 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:39857/solr&version=2&getVersions=100} status=0 QTime=21 
[junit4:junit4]   2> 6161 T108 C3 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6224 T108 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_2,generation=2,filenames=[_0_Asserting_0.tim, _0.fnm, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _0_Lucene41_0.pos, _0_nrm.cfs, _0_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _0_Lucene41_0.doc, _0_SimpleText_0.pst, _0_nrm.cfe, _0_MockVariableIntBlock_0.frq, _0_MockVariableIntBlock_0.pos, segments_2, _0.fdx, _0.si, _0_Asserting_0.pos, _0_Lucene41_0.tim, _0_MockVariableIntBlock_0.tii, _0.fdt, _0_Lucene41_0.tip, _0_Asserting_0.doc]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_3,generation=3,filenames=[_0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_Lucene41_0.doc, _0_Asserting_0.tip, _0_Lucene41_0.pos, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.skp, _1.fnm, _1_Asserting_0.tim, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_Asserting_0.tip, _1_MockVariableIntBlock_0.doc, _0_SimpleText_0.pst, _0_MockVariableIntBlock_0.pos, _0.si, _0_Lucene41_0.tim, _0_Asserting_0.doc, _0_Lucene41_0.tip, _0_MockVariableIntBlock_0.tii, _1_Asserting_0.pos, _1_nrm.cfs, _0.fnm, _1_MockVariableIntBlock_0.tii, _1_Lucene41_0.tim, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tip, _0_nrm.cfs, _1_MockVariableIntBlock_0.tib, _1_nrm.cfe, _1_Lucene41_0.pos, _1_MockVariableIntBlock_0.frq, _1.fdx, _1.fdt, _0_Lucene41_0.doc, _1.si, _0_MockVariableIntBlock_0.frq, _0_nrm.cfe, _0.fdx, _0_Asserting_0.pos, segments_3, _0.fdt, _1_SimpleText_0.pst]
[junit4:junit4]   2> 6225 T108 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 6233 T108 C3 oass.SolrIndexSearcher.<init> Opening Searcher@79b7d13e main
[junit4:junit4]   2> 6233 T108 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6234 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79b7d13e main{StandardDirectoryReader(segments_3:5 _0(4.1):C1 _1(4.1):C1)}
[junit4:junit4]   2> 6234 T108 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6236 T108 C3 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 75
[junit4:junit4]   2> 6241 T131 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6255 T131 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_2,generation=2,filenames=[_0_Asserting_0.tim, _0.fnm, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _0_Lucene41_0.pos, _0_nrm.cfs, _0_MockVariableIntBlock_0.skp, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _0_Lucene41_0.doc, _0_SimpleText_0.pst, _0_nrm.cfe, _0_MockVariableIntBlock_0.frq, _0_MockVariableIntBlock_0.pos, segments_2, _0.fdx, _0.si, _0_Asserting_0.pos, _0_Lucene41_0.tim, _0_MockVariableIntBlock_0.tii, _0.fdt, _0_Lucene41_0.tip, _0_Asserting_0.doc]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_3,generation=3,filenames=[_0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_Lucene41_0.doc, _0_Asserting_0.tip, _0_Lucene41_0.pos, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.skp, _1.fnm, _1_Asserting_0.tim, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_Asserting_0.tip, _1_MockVariableIntBlock_0.doc, _0_SimpleText_0.pst, _0_MockVariableIntBlock_0.pos, _0.si, _0_Lucene41_0.tim, _0_Asserting_0.doc, _0_Lucene41_0.tip, _0_MockVariableIntBlock_0.tii, _1_Asserting_0.pos, _1_nrm.cfs, _0.fnm, _1_MockVariableIntBlock_0.tii, _1_Lucene41_0.tim, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tip, _0_nrm.cfs, _1_MockVariableIntBlock_0.tib, _1_nrm.cfe, _1_Lucene41_0.pos, _1_MockVariableIntBlock_0.frq, _1.fdx, _1.fdt, _0_Lucene41_0.doc, _1.si, _0_MockVariableIntBlock_0.frq, _0_nrm.cfe, _0.fdx, _0_Asserting_0.pos, segments_3, _0.fdt, _1_SimpleText_0.pst]
[junit4:junit4]   2> 6256 T131 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 6264 T131 C4 oass.SolrIndexSearcher.<init> Opening Searcher@1d209d56 main
[junit4:junit4]   2> 6264 T131 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6265 T131 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6265 T121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d209d56 main{StandardDirectoryReader(segments_3:5 _0(4.1):C1 _1(4.1):C1)}
[junit4:junit4]   2> 6266 T131 C4 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 25
[junit4:junit4]   2> 6270 T106 C3 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=2 status=0 QTime=0 
[junit4:junit4]   2> 6275 T120 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 6287 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=6 
[junit4:junit4]   2> 6296 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 6305 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 6315 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 6324 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 6333 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 6342 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 6351 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 6356 T131 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:39857/solr] nUpdates=100
[junit4:junit4]   2> 6360 T108 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=0 
[junit4:junit4]   2> 6361 T131 C4 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 10 versions from 127.0.0.1:39857/solr
[junit4:junit4]   2> 6362 T131 C4 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:39857/solrn=8 versions=[10, 9, 8, 7, 6, 5, 4, 3]
[junit4:junit4]   2> 6366 T107 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=10,9,8,7,6,5,4,3&qt=/get&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6402 T131 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 6402 T131 C4 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:39857/solr&version=2&getVersions=100} status=0 QTime=47 
[junit4:junit4]   2> 6407 T106 C3 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6437 T106 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_3,generation=3,filenames=[_0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_Lucene41_0.doc, _0_Asserting_0.tip, _0_Lucene41_0.pos, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.skp, _1.fnm, _1_Asserting_0.tim, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_Asserting_0.tip, _1_MockVariableIntBlock_0.doc, _0_SimpleText_0.pst, _0_MockVariableIntBlock_0.pos, _0.si, _0_Lucene41_0.tim, _0_Asserting_0.doc, _0_Lucene41_0.tip, _0_MockVariableIntBlock_0.tii, _1_Asserting_0.pos, _1_nrm.cfs, _0.fnm, _1_MockVariableIntBlock_0.tii, _1_Lucene41_0.tim, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tip, _0_nrm.cfs, _1_MockVariableIntBlock_0.tib, _1_nrm.cfe, _1_Lucene41_0.pos, _1_MockVariableIntBlock_0.frq, _1.fdx, _1.fdt, _0_Lucene41_0.doc, _1.si, _0_MockVariableIntBlock_0.frq, _0_nrm.cfe, _0.fdx, _0_Asserting_0.pos, segments_3, _0.fdt, _1_SimpleText_0.pst]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_4,generation=4,filenames=[_2_Lucene41_0.pos, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _2_SimpleText_0.pst, _2_Asserting_0.pos, _0_Lucene41_0.pos, _0_MockVariableIntBlock_0.skp, _2_Lucene41_0.tip, _1.fnm, _1_Asserting_0.tim, _1_Asserting_0.tip, _0_SimpleText_0.pst, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0_Asserting_0.doc, _2_MockVariableIntBlock_0.pos, _1_Asserting_0.pos, _0.fnm, _1_MockVariableIntBlock_0.tii, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _1_MockVariableIntBlock_0.tib, _2_Asserting_0.tim, _1.fdx, _1_Lucene41_0.pos, _2_Asserting_0.tip, _0_Lucene41_0.doc, _1.fdt, _0_MockVariableIntBlock_0.frq, _2_MockVariableIntBlock_0.skp, _1_SimpleText_0.pst, _1_MockVariableIntBlock_0.skp, _0_Asserting_0.tim, _2.si, _2_MockVariableIntBlock_0.pyl, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _1_MockVariableIntBlock_0.pyl, _2_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pyl, _2_nrm.cfe, _0_MockVariableIntBlock_0.tib, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pos, _2_nrm.cfs, _0.si, _0_MockVariableIntBlock_0.tii, _1_nrm.cfs, _2_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _2.fnm, _1_nrm.cfe, _1_MockVariableIntBlock_0.frq, _1.si, _0_nrm.cfe, _0.fdx, _0_Asserting_0.pos, _0.fdt, segments_4]
[junit4:junit4]   2> 6438 T106 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 6447 T106 C3 oass.SolrIndexSearcher.<init> Opening Searcher@5449ff3f main
[junit4:junit4]   2> 6448 T106 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6449 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5449ff3f main{StandardDirectoryReader(segments_4:7 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8)}
[junit4:junit4]   2> 6449 T106 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6450 T106 C3 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 43
[junit4:junit4]   2> 6454 T131 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6533 T131 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_3,generation=3,filenames=[_0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _1_Lucene41_0.doc, _0_Asserting_0.tip, _0_Lucene41_0.pos, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _1_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.skp, _1.fnm, _1_Asserting_0.tim, _0_MockVariableIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_Asserting_0.tip, _1_MockVariableIntBlock_0.doc, _0_SimpleText_0.pst, _0_MockVariableIntBlock_0.pos, _0.si, _0_Lucene41_0.tim, _0_Asserting_0.doc, _0_Lucene41_0.tip, _0_MockVariableIntBlock_0.tii, _1_Asserting_0.pos, _1_nrm.cfs, _0.fnm, _1_MockVariableIntBlock_0.tii, _1_Lucene41_0.tim, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tip, _0_nrm.cfs, _1_MockVariableIntBlock_0.tib, _1_nrm.cfe, _1_Lucene41_0.pos, _1_MockVariableIntBlock_0.frq, _1.fdx, _1.fdt, _0_Lucene41_0.doc, _1.si, _0_MockVariableIntBlock_0.frq, _0_nrm.cfe, _0.fdx, _0_Asserting_0.pos, segments_3, _0.fdt, _1_SimpleText_0.pst]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_4,generation=4,filenames=[_2_Lucene41_0.pos, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _2_SimpleText_0.pst, _2_Asserting_0.pos, _0_Lucene41_0.pos, _0_MockVariableIntBlock_0.skp, _2_Lucene41_0.tip, _1.fnm, _1_Asserting_0.tim, _1_Asserting_0.tip, _0_SimpleText_0.pst, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0_Asserting_0.doc, _2_MockVariableIntBlock_0.pos, _1_Asserting_0.pos, _0.fnm, _1_MockVariableIntBlock_0.tii, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _1_MockVariableIntBlock_0.tib, _2_Asserting_0.tim, _1.fdx, _1_Lucene41_0.pos, _2_Asserting_0.tip, _0_Lucene41_0.doc, _1.fdt, _0_MockVariableIntBlock_0.frq, _2_MockVariableIntBlock_0.skp, _1_SimpleText_0.pst, _1_MockVariableIntBlock_0.skp, _0_Asserting_0.tim, _2.si, _2_MockVariableIntBlock_0.pyl, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _1_MockVariableIntBlock_0.pyl, _2_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pyl, _2_nrm.cfe, _0_MockVariableIntBlock_0.tib, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pos, _2_nrm.cfs, _0.si, _0_MockVariableIntBlock_0.tii, _1_nrm.cfs, _2_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _2.fnm, _1_nrm.cfe, _1_MockVariableIntBlock_0.frq, _1.si, _0_nrm.cfe, _0.fdx, _0_Asserting_0.pos, _0.fdt, segments_4]
[junit4:junit4]   2> 6534 T131 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 6543 T131 C4 oass.SolrIndexSearcher.<init> Opening Searcher@3f66cb16 main
[junit4:junit4]   2> 6543 T131 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6544 T131 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6544 T121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f66cb16 main{StandardDirectoryReader(segments_4:7 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8)}
[junit4:junit4]   2> 6545 T131 C4 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 91
[junit4:junit4]   2> 6550 T107 C3 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=10 status=0 QTime=1 
[junit4:junit4]   2> 6558 T120 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=10 status=0 QTime=1 
[junit4:junit4]   2> 6571 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 6578 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6584 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6590 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6595 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6600 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6606 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6611 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6617 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6623 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6628 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6635 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6641 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6648 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6654 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6659 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6665 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6669 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6672 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6677 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6682 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6687 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6691 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6696 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6700 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6705 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6709 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6714 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6719 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6724 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6730 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6735 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6740 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6745 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6750 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6756 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6761 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6766 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6771 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6776 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6780 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6784 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6788 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6792 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6796 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6800 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6805 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6810 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6814 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6818 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6822 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6826 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6829 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6834 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6839 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6845 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6850 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6855 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6860 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6864 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6868 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6874 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6878 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6882 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6886 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6891 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6894 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6898 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6903 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6908 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6913 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6918 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6923 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6929 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6934 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6939 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6955 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=11 
[junit4:junit4]   2> 6960 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6965 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6971 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6976 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6981 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6986 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6992 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6997 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7002 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7007 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7012 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7017 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7023 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7028 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7033 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7038 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7043 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7048 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7051 T131 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:39857/solr] nUpdates=100
[junit4:junit4]   2> 7055 T106 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 7056 T131 C4 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:39857/solr
[junit4:junit4]   2> 7057 T131 C4 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Our versions are too old. ourHighThreshold=8 otherLowThreshold=25
[junit4:junit4]   2> 7057 T131 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync failed
[junit4:junit4]   2> 7058 T131 C4 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:39857/solr&version=2&getVersions=100} status=0 QTime=7 
[junit4:junit4]   2> 7067 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 7072 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7077 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7082 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7087 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7092 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7097 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7102 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7107 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7112 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7117 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7122 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7127 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7132 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7137 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7142 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7147 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7152 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7157 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7169 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=9 
[junit4:junit4]   2> 7174 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7179 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7184 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7189 T131 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7194 T120 C4 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7198 T131 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:39857/solr] nUpdates=100
[junit4:junit4]   2> 7201 T108 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 7202 T131 C4 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:39857/solr
[junit4:junit4]   2> 7203 T131 C4 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:39857/solrn=70 versions=[105, 104, 103, 102, 101, 100, 99, 98, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 79, 78, 77, 76, 75, 74, 73, 72, 71, 70, 69, 68, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 37, 36]
[junit4:junit4]   2> 7210 T107 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=105,104,103,102,101,100,99,98,97,96,95,94,93,92,91,90,89,88,87,86,85,84,83,82,81,80,79,78,77,76,75,74,73,72,71,70,69,68,67,66,65,64,63,62,61,60,59,58,57,56,55,54,53,52,51,50,49,48,47,46,45,44,43,42,41,40,39,38,37,36&qt=/get&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 7266 T131 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 7267 T131 C4 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:39857/solr&version=2&getVersions=100} status=0 QTime=70 
[junit4:junit4]   2> 7271 T106 C3 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 7351 T106 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_4,generation=4,filenames=[_2_Lucene41_0.pos, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _2_SimpleText_0.pst, _2_Asserting_0.pos, _0_Lucene41_0.pos, _0_MockVariableIntBlock_0.skp, _2_Lucene41_0.tip, _1.fnm, _1_Asserting_0.tim, _1_Asserting_0.tip, _0_SimpleText_0.pst, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0_Asserting_0.doc, _2_MockVariableIntBlock_0.pos, _1_Asserting_0.pos, _0.fnm, _1_MockVariableIntBlock_0.tii, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _1_MockVariableIntBlock_0.tib, _2_Asserting_0.tim, _1.fdx, _1_Lucene41_0.pos, _2_Asserting_0.tip, _0_Lucene41_0.doc, _1.fdt, _0_MockVariableIntBlock_0.frq, _2_MockVariableIntBlock_0.skp, _1_SimpleText_0.pst, _1_MockVariableIntBlock_0.skp, _0_Asserting_0.tim, _2.si, _2_MockVariableIntBlock_0.pyl, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _1_MockVariableIntBlock_0.pyl, _2_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pyl, _2_nrm.cfe, _0_MockVariableIntBlock_0.tib, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pos, _2_nrm.cfs, _0.si, _0_MockVariableIntBlock_0.tii, _1_nrm.cfs, _2_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _2.fnm, _1_nrm.cfe, _1_MockVariableIntBlock_0.frq, _1.si, _0_nrm.cfe, _0.fdx, _0_Asserting_0.pos, _0.fdt, segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_5,generation=5,filenames=[_3_nrm.cfe, _2_Lucene41_0.pos, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _2_SimpleText_0.pst, _2_Asserting_0.pos, _3_MockVariableIntBlock_0.frq, _0_Lucene41_0.pos, _3.fdt, _0_MockVariableIntBlock_0.skp, _2_Lucene41_0.tip, _1.fnm, _1_Asserting_0.tim, _3_Lucene41_0.pos, _3_SimpleText_0.pst, _1_Asserting_0.tip, _0_SimpleText_0.pst, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0_Asserting_0.doc, _2_MockVariableIntBlock_0.pos, _1_Asserting_0.pos, _0.fnm, _1_MockVariableIntBlock_0.tii, _3_MockVariableIntBlock_0.doc, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _1_MockVariableIntBlock_0.tib, _3_nrm.cfs, _2_Asserting_0.tim, _1.fdx, _1_Lucene41_0.pos, _3_Lucene41_0.doc, _2_Asserting_0.tip, _0_Lucene41_0.doc, _1.fdt, _0_MockVariableIntBlock_0.frq, _3_MockVariableIntBlock_0.tib, _3.si, _2_MockVariableIntBlock_0.skp, _1_SimpleText_0.pst, _1_MockVariableIntBlock_0.skp, _0_Asserting_0.tim, _2.si, _2_MockVariableIntBlock_0.pyl, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _1_MockVariableIntBlock_0.pyl, _3.fnm, _3_Asserting_0.pos, _3_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pyl, _2_nrm.cfe, _0_MockVariableIntBlock_0.tib, _3_Lucene41_0.tip, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pos, _3_Lucene41_0.tim, _2_nrm.cfs, _0.si, _0_MockVariableIntBlock_0.tii, _1_nrm.cfs, _2_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _2.fnm, _1_nrm.cfe, _3.fdx, _1_MockVariableIntBlock_0.frq, _3_Asserting_0.tim, _3_Asserting_0.tip, _3_Asserting_0.doc, _1.si, _0_nrm.cfe, _3_MockVariableIntBlock_0.pyl, _0.fdx, _0_Asserting_0.pos, segments_5, _3_MockVariableIntBlock_0.skp, _0.fdt, _3_MockVariableIntBlock_0.pos]
[junit4:junit4]   2> 7352 T106 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 5
[junit4:junit4]   2> 7358 T106 C3 oass.SolrIndexSearcher.<init> Opening Searcher@62af9d74 main
[junit4:junit4]   2> 7358 T106 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7359 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@62af9d74 main{StandardDirectoryReader(segments_5:9 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95)}
[junit4:junit4]   2> 7360 T106 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 7360 T106 C3 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 89
[junit4:junit4]   2> 7364 T120 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 7394 T120 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_4,generation=4,filenames=[_2_Lucene41_0.pos, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _2_SimpleText_0.pst, _2_Asserting_0.pos, _0_Lucene41_0.pos, _0_MockVariableIntBlock_0.skp, _2_Lucene41_0.tip, _1.fnm, _1_Asserting_0.tim, _1_Asserting_0.tip, _0_SimpleText_0.pst, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0_Asserting_0.doc, _2_MockVariableIntBlock_0.pos, _1_Asserting_0.pos, _0.fnm, _1_MockVariableIntBlock_0.tii, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _1_MockVariableIntBlock_0.tib, _2_Asserting_0.tim, _1.fdx, _1_Lucene41_0.pos, _2_Asserting_0.tip, _0_Lucene41_0.doc, _1.fdt, _0_MockVariableIntBlock_0.frq, _2_MockVariableIntBlock_0.skp, _1_SimpleText_0.pst, _1_MockVariableIntBlock_0.skp, _0_Asserting_0.tim, _2.si, _2_MockVariableIntBlock_0.pyl, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _1_MockVariableIntBlock_0.pyl, _2_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pyl, _2_nrm.cfe, _0_MockVariableIntBlock_0.tib, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pos, _2_nrm.cfs, _0.si, _0_MockVariableIntBlock_0.tii, _1_nrm.cfs, _2_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _2.fnm, _1_nrm.cfe, _1_MockVariableIntBlock_0.frq, _1.si, _0_nrm.cfe, _0.fdx, _0_Asserting_0.pos, _0.fdt, segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_5,generation=5,filenames=[_3_nrm.cfe, _2_Lucene41_0.pos, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _2_SimpleText_0.pst, _2_Asserting_0.pos, _3_MockVariableIntBlock_0.frq, _0_Lucene41_0.pos, _3.fdt, _0_MockVariableIntBlock_0.skp, _2_Lucene41_0.tip, _1.fnm, _1_Asserting_0.tim, _3_Lucene41_0.pos, _3_SimpleText_0.pst, _1_Asserting_0.tip, _0_SimpleText_0.pst, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0_Asserting_0.doc, _2_MockVariableIntBlock_0.pos, _1_Asserting_0.pos, _0.fnm, _1_MockVariableIntBlock_0.tii, _3_MockVariableIntBlock_0.doc, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _1_MockVariableIntBlock_0.tib, _3_nrm.cfs, _2_Asserting_0.tim, _1.fdx, _1_Lucene41_0.pos, _3_Lucene41_0.doc, _2_Asserting_0.tip, _0_Lucene41_0.doc, _1.fdt, _0_MockVariableIntBlock_0.frq, _3_MockVariableIntBlock_0.tib, _3.si, _2_MockVariableIntBlock_0.skp, _1_SimpleText_0.pst, _1_MockVariableIntBlock_0.skp, _0_Asserting_0.tim, _2.si, _2_MockVariableIntBlock_0.pyl, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _1_MockVariableIntBlock_0.pyl, _3.fnm, _3_Asserting_0.pos, _3_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pyl, _2_nrm.cfe, _0_MockVariableIntBlock_0.tib, _3_Lucene41_0.tip, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pos, _3_Lucene41_0.tim, _2_nrm.cfs, _0.si, _0_MockVariableIntBlock_0.tii, _1_nrm.cfs, _2_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _2.fnm, _1_nrm.cfe, _3.fdx, _1_MockVariableIntBlock_0.frq, _3_Asserting_0.tim, _3_Asserting_0.tip, _3_Asserting_0.doc, _1.si, _0_nrm.cfe, _3_MockVariableIntBlock_0.pyl, _0.fdx, _0_Asserting_0.pos, segments_5, _3_MockVariableIntBlock_0.skp, _0.fdt, _3_MockVariableIntBlock_0.pos]
[junit4:junit4]   2> 7395 T120 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 5
[junit4:junit4]   2> 7401 T120 C4 oass.SolrIndexSearcher.<init> Opening Searcher@47f07360 main
[junit4:junit4]   2> 7401 T120 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7402 T121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47f07360 main{StandardDirectoryReader(segments_5:9 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95)}
[junit4:junit4]   2> 7403 T120 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 7403 T120 C4 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 40
[junit4:junit4]   2> 7599 T108 C3 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=105 status=0 QTime=193 
[junit4:junit4]   2> 7605 T131 C4 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=105 status=0 QTime=2 
[junit4:junit4]   2> 7618 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=7 
[junit4:junit4]   2> 7623 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7806 T107 C3 oass.SolrIndexSearcher.<init> Opening Searcher@64b25680 realtime
[junit4:junit4]   2> 7807 T107 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7808 T107 C3 REQ [collection1] webapp=/solr path=/update params={_version_=-1003&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=182 
[junit4:junit4]   2> 7815 T106 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 7820 T107 C3 REQ [collection1] webapp=/solr path=/update params={_version_=-1005&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7824 T120 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:39857/solr] nUpdates=100
[junit4:junit4]   2> 7833 T108 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=5 
[junit4:junit4]   2> 7834 T120 C4 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:39857/solr
[junit4:junit4]   2> 7835 T120 C4 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:39857/solrn=5 versions=[-1005, 1004, -1003, 1002, 1001]
[junit4:junit4]   2> 7839 T106 C3 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=-1005,1004,-1003,1002,1001&qt=/get&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 7875 T120 C4 oass.SolrIndexSearcher.<init> Opening Searcher@79a6a3c3 realtime
[junit4:junit4]   2> 7876 T120 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7879 T120 C4 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 7879 T120 C4 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:39857/solr&version=2&getVersions=100} status=0 QTime=56 
[junit4:junit4]   2> 7883 T107 C3 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 7894 T107 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_5,generation=5,filenames=[_3_nrm.cfe, _2_Lucene41_0.pos, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _2_SimpleText_0.pst, _2_Asserting_0.pos, _3_MockVariableIntBlock_0.frq, _0_Lucene41_0.pos, _3.fdt, _0_MockVariableIntBlock_0.skp, _2_Lucene41_0.tip, _1.fnm, _1_Asserting_0.tim, _3_Lucene41_0.pos, _3_SimpleText_0.pst, _1_Asserting_0.tip, _0_SimpleText_0.pst, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0_Asserting_0.doc, _2_MockVariableIntBlock_0.pos, _1_Asserting_0.pos, _0.fnm, _1_MockVariableIntBlock_0.tii, _3_MockVariableIntBlock_0.doc, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _1_MockVariableIntBlock_0.tib, _3_nrm.cfs, _2_Asserting_0.tim, _1.fdx, _1_Lucene41_0.pos, _3_Lucene41_0.doc, _2_Asserting_0.tip, _0_Lucene41_0.doc, _1.fdt, _0_MockVariableIntBlock_0.frq, _3_MockVariableIntBlock_0.tib, _3.si, _2_MockVariableIntBlock_0.skp, _1_SimpleText_0.pst, _1_MockVariableIntBlock_0.skp, _0_Asserting_0.tim, _2.si, _2_MockVariableIntBlock_0.pyl, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _1_MockVariableIntBlock_0.pyl, _3.fnm, _3_Asserting_0.pos, _3_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pyl, _2_nrm.cfe, _0_MockVariableIntBlock_0.tib, _3_Lucene41_0.tip, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pos, _3_Lucene41_0.tim, _2_nrm.cfs, _0.si, _0_MockVariableIntBlock_0.tii, _1_nrm.cfs, _2_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _2.fnm, _1_nrm.cfe, _3.fdx, _1_MockVariableIntBlock_0.frq, _3_Asserting_0.tim, _3_Asserting_0.tip, _3_Asserting_0.doc, _1.si, _0_nrm.cfe, _3_MockVariableIntBlock_0.pyl, _0.fdx, _0_Asserting_0.pos, segments_5, _3_MockVariableIntBlock_0.skp, _0.fdt, _3_MockVariableIntBlock_0.pos]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1a4eb98b lockFactory=org.apache.lucene.store.NativeFSLockFactory@67ce08c7),segFN=segments_6,generation=6,filenames=[_2_Lucene41_0.pos, _5_nrm.cfe, _5_Lucene41_0.tim, _2_SimpleText_0.pst, _2_Asserting_0.pos, _5_Lucene41_0.tip, _3_MockVariableIntBlock_0.frq, _3.fdt, _1_Asserting_0.tim, _1_Asserting_0.tip, _0_SimpleText_0.pst, _5_MockVariableIntBlock_0.tii, _5_Asserting_0.pos, _1_MockVariableIntBlock_0.tii, _2.fdt, _2.fdx, _1_MockVariableIntBlock_0.tib, _1_Lucene41_0.pos, _1.fdx, _2_Asserting_0.tim, _2_Asserting_0.tip, _5_nrm.cfs, _1.fdt, _0_MockVariableIntBlock_0.frq, _3.si, _2_MockVariableIntBlock_0.skp, _1_SimpleText_0.pst, _2.si, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _2_MockVariableIntBlock_0.pyl, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _3.fnm, _3_Asserting_0.pos, _2_MockVariableIntBlock_0.doc, _3_MockVariableIntBlock_0.tii, _5_Asserting_0.doc, _0_MockVariableIntBlock_0.pyl, _2_nrm.cfe, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pos, _2_nrm.cfs, _2_MockVariableIntBlock_0.tii, _5_MockVariableIntBlock_0.tib, _2_MockVariableIntBlock_0.tib, _1_Lucene41_0.tim, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tip, _5_MockVariableIntBlock_0.frq, _2.fnm, _3.fdx, _1.si, _3_Asserting_0.doc, _3_MockVariableIntBlock_0.pyl, _0_Asserting_0.pos, _3_MockVariableIntBlock_0.pos, _3_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_MockVariableIntBlock_0.pyl, _5_Lucene41_0.pos, _0_Lucene41_0.pos, _0_MockVariableIntBlock_0.skp, _2_Lucene41_0.tip, _1.fnm, _3_Lucene41_0.pos, _3_SimpleText_0.pst, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Asserting_0.doc, _0_Lucene41_0.tip, _2_MockVariableIntBlock_0.pos, _1_Asserting_0.pos, _0.fnm, _5_MockVariableIntBlock_0.pos, _5_Asserting_0.tim, _3_MockVariableIntBlock_0.doc, _2_Lucene41_0.doc, _5.si, _3_nrm.cfs, _3_Lucene41_0.doc, _0_Lucene41_0.doc, _5_Asserting_0.tip, _3_MockVariableIntBlock_0.tib, _5_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _0_MockVariableIntBlock_0.tib, _5_MockVariableIntBlock_0.skp, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _0_MockVariableIntBlock_0.tii, _5_SimpleText_0.pst, _1_nrm.cfs, _5.fnm, _0_nrm.cfs, _1_nrm.cfe, _1_MockVariableIntBlock_0.frq, _3_Asserting_0.tim, _3_Asserting_0.tip, _0_nrm.cfe, _5.fdx, _0.fdx, segments_6, _3_MockVariableIntBlock_0.skp, _0.fdt]
[junit4:junit4]   2> 7895 T107 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 6
[junit4:junit4]   2> 7899 T107 C3 oass.SolrIndexSearcher.<init> Opening Searcher@fbf00a9 main
[junit4:junit4]   2> 7899 T107 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7901 T109 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fbf00a9 main{StandardDirectoryReader(segments_6:16:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1)}
[junit4:junit4]   2> 7901 T107 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 7903 T107 C3 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 20
[junit4:junit4]   2> 7907 T131 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 7919 T131 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_5,generation=5,filenames=[_3_nrm.cfe, _2_Lucene41_0.pos, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _2_SimpleText_0.pst, _2_Asserting_0.pos, _3_MockVariableIntBlock_0.frq, _0_Lucene41_0.pos, _3.fdt, _0_MockVariableIntBlock_0.skp, _2_Lucene41_0.tip, _1.fnm, _1_Asserting_0.tim, _3_Lucene41_0.pos, _3_SimpleText_0.pst, _1_Asserting_0.tip, _0_SimpleText_0.pst, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0_Asserting_0.doc, _2_MockVariableIntBlock_0.pos, _1_Asserting_0.pos, _0.fnm, _1_MockVariableIntBlock_0.tii, _3_MockVariableIntBlock_0.doc, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _1_MockVariableIntBlock_0.tib, _3_nrm.cfs, _2_Asserting_0.tim, _1.fdx, _1_Lucene41_0.pos, _3_Lucene41_0.doc, _2_Asserting_0.tip, _0_Lucene41_0.doc, _1.fdt, _0_MockVariableIntBlock_0.frq, _3_MockVariableIntBlock_0.tib, _3.si, _2_MockVariableIntBlock_0.skp, _1_SimpleText_0.pst, _1_MockVariableIntBlock_0.skp, _0_Asserting_0.tim, _2.si, _2_MockVariableIntBlock_0.pyl, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _1_MockVariableIntBlock_0.pyl, _3.fnm, _3_Asserting_0.pos, _3_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pyl, _2_nrm.cfe, _0_MockVariableIntBlock_0.tib, _3_Lucene41_0.tip, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pos, _3_Lucene41_0.tim, _2_nrm.cfs, _0.si, _0_MockVariableIntBlock_0.tii, _1_nrm.cfs, _2_MockVariableIntBlock_0.tii, _2_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _2.fnm, _1_nrm.cfe, _3.fdx, _1_MockVariableIntBlock_0.frq, _3_Asserting_0.tim, _3_Asserting_0.tip, _3_Asserting_0.doc, _1.si, _0_nrm.cfe, _3_MockVariableIntBlock_0.pyl, _0.fdx, _0_Asserting_0.pos, segments_5, _3_MockVariableIntBlock_0.skp, _0.fdt, _3_MockVariableIntBlock_0.pos]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7041825e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6e8d1800),segFN=segments_6,generation=6,filenames=[_2_Lucene41_0.pos, _5_nrm.cfe, _5_Lucene41_0.tim, _2_SimpleText_0.pst, _2_Asserting_0.pos, _5_Lucene41_0.tip, _3_MockVariableIntBlock_0.frq, _3.fdt, _1_Asserting_0.tim, _1_Asserting_0.tip, _0_SimpleText_0.pst, _5_MockVariableIntBlock_0.tii, _5_Asserting_0.pos, _1_MockVariableIntBlock_0.tii, _2.fdt, _2.fdx, _1_MockVariableIntBlock_0.tib, _1_Lucene41_0.pos, _1.fdx, _2_Asserting_0.tim, _2_Asserting_0.tip, _5_nrm.cfs, _1.fdt, _0_MockVariableIntBlock_0.frq, _3.si, _2_MockVariableIntBlock_0.skp, _1_SimpleText_0.pst, _2.si, _0_Asserting_0.tim, _1_MockVariableIntBlock_0.skp, _2_MockVariableIntBlock_0.pyl, _0_Asserting_0.tip, _0_MockVariableIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _3.fnm, _3_Asserting_0.pos, _2_MockVariableIntBlock_0.doc, _3_MockVariableIntBlock_0.tii, _5_Asserting_0.doc, _0_MockVariableIntBlock_0.pyl, _2_nrm.cfe, _2_Asserting_0.doc, _1_MockVariableIntBlock_0.doc, _0_MockVariableIntBlock_0.pos, _2_nrm.cfs, _2_MockVariableIntBlock_0.tii, _5_MockVariableIntBlock_0.tib, _2_MockVariableIntBlock_0.tib, _1_Lucene41_0.tim, _1_MockVariableIntBlock_0.pos, _1_Lucene41_0.tip, _5_MockVariableIntBlock_0.frq, _2.fnm, _3.fdx, _1.si, _3_Asserting_0.doc, _3_MockVariableIntBlock_0.pyl, _0_Asserting_0.pos, _3_MockVariableIntBlock_0.pos, _3_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_MockVariableIntBlock_0.pyl, _5_Lucene41_0.pos, _0_Lucene41_0.pos, _0_MockVariableIntBlock_0.skp, _2_Lucene41_0.tip, _1.fnm, _3_Lucene41_0.pos, _3_SimpleText_0.pst, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Asserting_0.doc, _0_Lucene41_0.tip, _2_MockVariableIntBlock_0.pos, _1_Asserting_0.pos, _0.fnm, _5_MockVariableIntBlock_0.pos, _5_Asserting_0.tim, _3_MockVariableIntBlock_0.doc, _2_Lucene41_0.doc, _5.si, _3_nrm.cfs, _3_Lucene41_0.doc, _0_Lucene41_0.doc, _5_Asserting_0.tip, _3_MockVariableIntBlock_0.tib, _5_MockVariableIntBlock_0.doc, _1_Asserting_0.doc, _2_MockVariableIntBlock_0.frq, _0_MockVariableIntBlock_0.tib, _5_MockVariableIntBlock_0.skp, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _0_MockVariableIntBlock_0.tii, _5_SimpleText_0.pst, _1_nrm.cfs, _5.fnm, _0_nrm.cfs, _1_nrm.cfe, _1_MockVariableIntBlock_0.frq, _3_Asserting_0.tim, _3_Asserting_0.tip, _0_nrm.cfe, _5.fdx, _0.fdx, segments_6, _3_MockVariableIntBlock_0.skp, _0.fdt]
[junit4:junit4]   2> 7920 T131 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 6
[junit4:junit4]   2> 7925 T131 C4 oass.SolrIndexSearcher.<init> Opening Searcher@5826d8e6 main
[junit4:junit4]   2> 7926 T131 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7929 T121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5826d8e6 main{StandardDirectoryReader(segments_6:16:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1)}
[junit4:junit4]   2> 7933 T131 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 7933 T131 C4 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 26
[junit4:junit4]   2> 7938 T106 C3 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=106 status=0 QTime=2 
[junit4:junit4]   2> 7945 T120 C4 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=106 status=0 QTime=3 
[junit4:junit4]   2> 7953 T108 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 7958 T107 C3 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7969 T106 C3 oass.SolrIndexSearcher.<init> Opening Searcher@61b2e165 realtime
[junit4:junit4]   2> 7970 T106 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper


[...truncated too long message...]

ept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=241, name=qtp2115930648-241 Acceptor2 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=176, name=qtp1257641869-176 Acceptor3 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    9) Thread[id=369, name=qtp1711596206-369 Acceptor0 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   10) Thread[id=370, name=qtp1711596206-370 Acceptor1 SelectChannelConnector@0.0.0.0:22970, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=221, name=qtp1229973302-221 Acceptor0 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=222, name=qtp1229973302-222 Acceptor1 SelectChannelConnector@0.0.0.0:39123, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=372, name=qtp1711596206-372 Acceptor3 SelectChannelConnector@0.0.0.0:22970, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=242, name=qtp2115930648-242 Acceptor3 SelectChannelConnector@0.0.0.0:65310, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=173, name=qtp1257641869-173 Acceptor0 SelectChannelConnector@0.0.0.0:39867, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=175, name=qtp1257641869-175 Acceptor2 SelectChannelConnector@0.0.0.0:39867, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([A461CFFBFF73C82E]:0)
[junit4:junit4] Completed on J0 in 237.38s, 1 test, 2 errors <<< FAILURES!

[...truncated 110 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 788 ignored

Total time: 21 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 831 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/831/

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test:     1) Thread[id=356, name=qtp1249338498-356 Acceptor0 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=341, name=qtp638227191-341 Acceptor3 SelectChannelConnector@0.0.0.0:21381, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=290, name=qtp445567561-290 Acceptor0 SelectChannelConnector@0.0.0.0:35206, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=359, name=qtp1249338498-359 Acceptor3 SelectChannelConnector@0.0.0.0:51985, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=339, name=qtp638227191-339 Acceptor1 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=293, name=qtp445567561-293 Acceptor3 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=291, name=qtp445567561-291 Acceptor1 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=309, name=qtp131312334-309 Acceptor1 SelectChannelConnector@0.0.0.0:18586, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=292, name=qtp445567561-292 Acceptor2 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=358, name=qtp1249338498-358 Acceptor2 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=310, name=qtp131312334-310 Acceptor2 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=357, name=qtp1249338498-357 Acceptor1 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=340, name=qtp638227191-340 Acceptor2 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=311, name=qtp131312334-311 Acceptor3 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=308, name=qtp131312334-308 Acceptor0 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=338, name=qtp638227191-338 Acceptor0 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test: 
   1) Thread[id=356, name=qtp1249338498-356 Acceptor0 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=341, name=qtp638227191-341 Acceptor3 SelectChannelConnector@0.0.0.0:21381, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=290, name=qtp445567561-290 Acceptor0 SelectChannelConnector@0.0.0.0:35206, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=359, name=qtp1249338498-359 Acceptor3 SelectChannelConnector@0.0.0.0:51985, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=339, name=qtp638227191-339 Acceptor1 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=293, name=qtp445567561-293 Acceptor3 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=291, name=qtp445567561-291 Acceptor1 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=309, name=qtp131312334-309 Acceptor1 SelectChannelConnector@0.0.0.0:18586, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=292, name=qtp445567561-292 Acceptor2 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=358, name=qtp1249338498-358 Acceptor2 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=310, name=qtp131312334-310 Acceptor2 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=357, name=qtp1249338498-357 Acceptor1 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=340, name=qtp638227191-340 Acceptor2 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=311, name=qtp131312334-311 Acceptor3 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=308, name=qtp131312334-308 Acceptor0 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=338, name=qtp638227191-338 Acceptor0 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([68B5074705293DC3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=356, name=qtp1249338498-356 Acceptor0 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=290, name=qtp445567561-290 Acceptor0 SelectChannelConnector@0.0.0.0:35206, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=341, name=qtp638227191-341 Acceptor3 SelectChannelConnector@0.0.0.0:21381, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=359, name=qtp1249338498-359 Acceptor3 SelectChannelConnector@0.0.0.0:51985, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=339, name=qtp638227191-339 Acceptor1 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=293, name=qtp445567561-293 Acceptor3 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=291, name=qtp445567561-291 Acceptor1 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=309, name=qtp131312334-309 Acceptor1 SelectChannelConnector@0.0.0.0:18586, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=292, name=qtp445567561-292 Acceptor2 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=358, name=qtp1249338498-358 Acceptor2 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=310, name=qtp131312334-310 Acceptor2 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=357, name=qtp1249338498-357 Acceptor1 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=340, name=qtp638227191-340 Acceptor2 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=311, name=qtp131312334-311 Acceptor3 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=308, name=qtp131312334-308 Acceptor0 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=338, name=qtp638227191-338 Acceptor0 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=356, name=qtp1249338498-356 Acceptor0 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=290, name=qtp445567561-290 Acceptor0 SelectChannelConnector@0.0.0.0:35206, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=341, name=qtp638227191-341 Acceptor3 SelectChannelConnector@0.0.0.0:21381, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=359, name=qtp1249338498-359 Acceptor3 SelectChannelConnector@0.0.0.0:51985, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=339, name=qtp638227191-339 Acceptor1 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=293, name=qtp445567561-293 Acceptor3 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=291, name=qtp445567561-291 Acceptor1 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=309, name=qtp131312334-309 Acceptor1 SelectChannelConnector@0.0.0.0:18586, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=292, name=qtp445567561-292 Acceptor2 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=358, name=qtp1249338498-358 Acceptor2 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=310, name=qtp131312334-310 Acceptor2 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=357, name=qtp1249338498-357 Acceptor1 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=340, name=qtp638227191-340 Acceptor2 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=311, name=qtp131312334-311 Acceptor3 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=308, name=qtp131312334-308 Acceptor0 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=338, name=qtp638227191-338 Acceptor0 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([68B5074705293DC3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedQueryElevationComponentTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedQueryElevationComponentTest:     1) Thread[id=15, name=qtp485367518-15 Acceptor1 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=45, name=qtp810158789-45 Acceptor2 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=34, name=qtp1912354856-34 Acceptor3 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=14, name=qtp485367518-14 Acceptor0 SelectChannelConnector@0.0.0.0:15953, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=44, name=qtp810158789-44 Acceptor1 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=60, name=qtp1329873750-60 Acceptor3 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=19, name=qtp485367518-19 Acceptor2 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=58, name=qtp1329873750-58 Acceptor1 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=31, name=qtp1912354856-31 Acceptor0 SelectChannelConnector@0.0.0.0:17757, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=33, name=qtp1912354856-33 Acceptor2 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=59, name=qtp1329873750-59 Acceptor2 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=43, name=qtp810158789-43 Acceptor0 SelectChannelConnector@0.0.0.0:57549, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=18, name=qtp485367518-18 Acceptor3 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=32, name=qtp1912354856-32 Acceptor1 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=46, name=qtp810158789-46 Acceptor3 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=57, name=qtp1329873750-57 Acceptor0 SelectChannelConnector@0.0.0.0:56935, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedQueryElevationComponentTest: 
   1) Thread[id=15, name=qtp485367518-15 Acceptor1 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=45, name=qtp810158789-45 Acceptor2 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=34, name=qtp1912354856-34 Acceptor3 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=14, name=qtp485367518-14 Acceptor0 SelectChannelConnector@0.0.0.0:15953, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=44, name=qtp810158789-44 Acceptor1 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=60, name=qtp1329873750-60 Acceptor3 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=19, name=qtp485367518-19 Acceptor2 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=58, name=qtp1329873750-58 Acceptor1 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=31, name=qtp1912354856-31 Acceptor0 SelectChannelConnector@0.0.0.0:17757, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=33, name=qtp1912354856-33 Acceptor2 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=59, name=qtp1329873750-59 Acceptor2 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=43, name=qtp810158789-43 Acceptor0 SelectChannelConnector@0.0.0.0:57549, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=18, name=qtp485367518-18 Acceptor3 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=32, name=qtp1912354856-32 Acceptor1 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=46, name=qtp810158789-46 Acceptor3 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=57, name=qtp1329873750-57 Acceptor0 SelectChannelConnector@0.0.0.0:56935, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([68B5074705293DC3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedQueryElevationComponentTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=15, name=qtp485367518-15 Acceptor1 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=14, name=qtp485367518-14 Acceptor0 SelectChannelConnector@0.0.0.0:15953, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=34, name=qtp1912354856-34 Acceptor3 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=45, name=qtp810158789-45 Acceptor2 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=44, name=qtp810158789-44 Acceptor1 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=60, name=qtp1329873750-60 Acceptor3 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=19, name=qtp485367518-19 Acceptor2 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=31, name=qtp1912354856-31 Acceptor0 SelectChannelConnector@0.0.0.0:17757, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=58, name=qtp1329873750-58 Acceptor1 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=33, name=qtp1912354856-33 Acceptor2 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=59, name=qtp1329873750-59 Acceptor2 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=43, name=qtp810158789-43 Acceptor0 SelectChannelConnector@0.0.0.0:57549, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=18, name=qtp485367518-18 Acceptor3 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=32, name=qtp1912354856-32 Acceptor1 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=46, name=qtp810158789-46 Acceptor3 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=57, name=qtp1329873750-57 Acceptor0 SelectChannelConnector@0.0.0.0:56935, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=15, name=qtp485367518-15 Acceptor1 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=14, name=qtp485367518-14 Acceptor0 SelectChannelConnector@0.0.0.0:15953, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=34, name=qtp1912354856-34 Acceptor3 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=45, name=qtp810158789-45 Acceptor2 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=44, name=qtp810158789-44 Acceptor1 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=60, name=qtp1329873750-60 Acceptor3 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=19, name=qtp485367518-19 Acceptor2 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=31, name=qtp1912354856-31 Acceptor0 SelectChannelConnector@0.0.0.0:17757, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=58, name=qtp1329873750-58 Acceptor1 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=33, name=qtp1912354856-33 Acceptor2 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=59, name=qtp1329873750-59 Acceptor2 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=43, name=qtp810158789-43 Acceptor0 SelectChannelConnector@0.0.0.0:57549, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=18, name=qtp485367518-18 Acceptor3 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=32, name=qtp1912354856-32 Acceptor1 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=46, name=qtp810158789-46 Acceptor3 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=57, name=qtp1329873750-57 Acceptor0 SelectChannelConnector@0.0.0.0:56935, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([68B5074705293DC3]:0)


FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
Shard still reported as live in zk

Stack Trace:
java.lang.AssertionError: Shard still reported as live in zk
	at __randomizedtesting.SeedInfo.seed([68B5074705293DC3:E953895F72765DFF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitToSeeNotLive(AbstractFullDistribZkTestBase.java:1218)
	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:190)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:83)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:695)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 7626 lines...]
[junit4:junit4] Suite: org.apache.solr.handler.component.DistributedQueryElevationComponentTest
[junit4:junit4]   2> 299 T10 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 940 T10 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1286 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:15953
[junit4:junit4]   2> 1491 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1492 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1493 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2280 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2284 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2284 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2314 T10 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2319 T10 oasc.CoreContainer.<init> New CoreContainer 2001431298
[junit4:junit4]   2> 2320 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2320 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2403 T10 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2487 T10 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2488 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2490 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2491 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2591 T10 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2770 T10 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-elevate.xml
[junit4:junit4]   2> 2798 T10 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2825 T10 oass.IndexSchema.readSchema Schema name=example
[junit4:junit4]   2> 3861 T10 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 3864 T10 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 4236 T10 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/control/data/
[junit4:junit4]   2> 4240 T10 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@29a220e6
[junit4:junit4]   2> 4419 T10 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/control/data/index/
[junit4:junit4]   2> 4421 T10 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 4492 T10 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/control/data/index forceNew:false
[junit4:junit4]   2> 4779 T10 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@135d4273 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@778abc7b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4779 T10 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5575 T10 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5584 T10 oasc.RequestHandlers.initHandlersFromConfig created /elevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5585 T10 oasc.RequestHandlers.initHandlersFromConfig created /dataElevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5704 T10 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5724 T10 oasc.RequestHandlers.initHandlersFromConfig WARNING no default request handler is registered (either '/select' or 'standard')
[junit4:junit4]   2> 5728 T10 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 5736 T10 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-MINIMAL.XML, TRIVIAL.UPDATEPROCESSOR1.JS, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SCHEMA-REPLICATION1.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA_CODEC.XML, SCHEMA.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, SOLRCONFIG-FUNCTIONQUERY.XML, STOP-2.TXT, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 5798 T10 oass.SolrIndexSearcher.<init> Opening Searcher@3c6b93c5 main
[junit4:junit4]   2> 5799 T10 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5864 T10 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5865 T10 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5878 T10 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 5930 T10 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 6035 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 6036 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 6036 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 6037 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 6037 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 6038 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 6038 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 6039 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 6040 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 6074 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 6494 T10 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 6495 T10 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 6496 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 6496 T25 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3c6b93c5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6550 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 6570 T10 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 6573 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:17757
[junit4:junit4]   2> 6573 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6574 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 6575 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 6605 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 6605 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6606 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 6606 T10 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 6607 T10 oasc.CoreContainer.<init> New CoreContainer 1592409311
[junit4:junit4]   2> 6608 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 6608 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 6633 T10 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 6656 T10 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 6657 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 6658 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 6659 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 6712 T10 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 6784 T10 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-elevate.xml
[junit4:junit4]   2> 6785 T10 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 6805 T10 oass.IndexSchema.readSchema Schema name=example
[junit4:junit4]   2> 7023 T10 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 7026 T10 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 7038 T10 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard0/data/
[junit4:junit4]   2> 7038 T10 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@29a220e6
[junit4:junit4]   2> 7040 T10 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard0/data/index/
[junit4:junit4]   2> 7041 T10 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 7042 T10 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard0/data/index forceNew:false
[junit4:junit4]   2> 7044 T10 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@20b00ec2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fbb8694; maxCacheMB=0.7099609375 maxMergeSizeMB=0.041015625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 7044 T10 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 7048 T10 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 7049 T10 oasc.RequestHandlers.initHandlersFromConfig created /elevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7057 T10 oasc.RequestHandlers.initHandlersFromConfig created /dataElevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7062 T10 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 7063 T10 oasc.RequestHandlers.initHandlersFromConfig WARNING no default request handler is registered (either '/select' or 'standard')
[junit4:junit4]   2> 7064 T10 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 7067 T10 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-MINIMAL.XML, TRIVIAL.UPDATEPROCESSOR1.JS, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SCHEMA-REPLICATION1.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA_CODEC.XML, SCHEMA.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, SOLRCONFIG-FUNCTIONQUERY.XML, STOP-2.TXT, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 7069 T10 oass.SolrIndexSearcher.<init> Opening Searcher@2a47b35d main
[junit4:junit4]   2> 7070 T10 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7070 T10 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 7071 T10 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7073 T10 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 7085 T10 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 7097 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 7098 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 7098 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 7099 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 7099 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 7100 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 7100 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 7101 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 7101 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 7102 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 7119 T10 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 7120 T10 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 7120 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 7121 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 7124 T37 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2a47b35d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 7130 T10 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 7133 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:57549
[junit4:junit4]   2> 7134 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7134 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 7135 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7148 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 7148 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7149 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 7149 T10 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 7150 T10 oasc.CoreContainer.<init> New CoreContainer 1085837532
[junit4:junit4]   2> 7150 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7151 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7168 T10 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 7184 T10 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 7185 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 7186 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 7187 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 7226 T10 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 7274 T10 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-elevate.xml
[junit4:junit4]   2> 7275 T10 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 7290 T10 oass.IndexSchema.readSchema Schema name=example
[junit4:junit4]   2> 7479 T10 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 7482 T10 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 7492 T10 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard1/data/
[junit4:junit4]   2> 7493 T10 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@29a220e6
[junit4:junit4]   2> 7494 T10 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard1/data/index/
[junit4:junit4]   2> 7495 T10 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 7496 T10 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard1/data/index forceNew:false
[junit4:junit4]   2> 7498 T10 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@67a5fb5a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@79123c5f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 7498 T10 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 7502 T10 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 7503 T10 oasc.RequestHandlers.initHandlersFromConfig created /elevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7505 T10 oasc.RequestHandlers.initHandlersFromConfig created /dataElevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7506 T10 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 7507 T10 oasc.RequestHandlers.initHandlersFromConfig WARNING no default request handler is registered (either '/select' or 'standard')
[junit4:junit4]   2> 7508 T10 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 7510 T10 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-MINIMAL.XML, TRIVIAL.UPDATEPROCESSOR1.JS, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SCHEMA-REPLICATION1.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA_CODEC.XML, SCHEMA.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, SOLRCONFIG-FUNCTIONQUERY.XML, STOP-2.TXT, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 7512 T10 oass.SolrIndexSearcher.<init> Opening Searcher@74021a1e main
[junit4:junit4]   2> 7513 T10 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7514 T10 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 7514 T10 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7516 T10 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 7529 T10 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 7542 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 7543 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 7543 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 7543 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 7544 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 7544 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 7545 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 7545 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 7546 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 7546 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 7567 T10 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 7568 T10 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 7568 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 7569 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 7570 T47 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@74021a1e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 7579 T10 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 7581 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:56935
[junit4:junit4]   2> 7582 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7583 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 7583 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7596 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 7597 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7597 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 7598 T10 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 7598 T10 oasc.CoreContainer.<init> New CoreContainer 144310161
[junit4:junit4]   2> 7599 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7599 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7616 T10 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 7633 T10 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 7633 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 7634 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 7635 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 7671 T10 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 7717 T10 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-elevate.xml
[junit4:junit4]   2> 7718 T10 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 7733 T10 oass.IndexSchema.readSchema Schema name=example
[junit4:junit4]   2> 7925 T10 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 7927 T10 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 7938 T10 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard2/data/
[junit4:junit4]   2> 7938 T10 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@29a220e6
[junit4:junit4]   2> 7940 T10 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard2/data/index/
[junit4:junit4]   2> 7941 T10 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 7941 T10 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.handler.component.DistributedQueryElevationComponentTest-1351856201674/shard2/data/index forceNew:false
[junit4:junit4]   2> 7943 T10 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6b0cc9b4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17496e57),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 7943 T10 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 7947 T10 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 7948 T10 oasc.RequestHandlers.initHandlersFromConfig created /elevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7949 T10 oasc.RequestHandlers.initHandlersFromConfig created /dataElevate: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7951 T10 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 7951 T10 oasc.RequestHandlers.initHandlersFromConfig WARNING no default request handler is registered (either '/select' or 'standard')
[junit4:junit4]   2> 7953 T10 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 7955 T10 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-MINIMAL.XML, TRIVIAL.UPDATEPROCESSOR1.JS, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-REPLICATION2.XML, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SCHEMA-REPLICATION1.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SOLRCONFIG-SPELLCHECKER.XML, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SCHEMA-XINCLUDE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA_CODEC.XML, SCHEMA.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, SOLRCONFIG-FUNCTIONQUERY.XML, STOP-2.TXT, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-SLAVE1.XML, SOLRCONFIG-DELPOLICY1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 7957 T10 oass.SolrIndexSearcher.<init> Opening Searcher@49b9ef36 main
[junit4:junit4]   2> 7957 T10 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7958 T10 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 7958 T10 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7960 T10 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 7972 T10 oashc.QueryElevationComponent.inform Loading QueryElevation from: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/elevate.xml
[junit4:junit4]   2> 7984 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 7984 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 7985 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 7985 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 7986 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 7986 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 7987 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 7987 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 7988 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 7988 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 8002 T10 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 8002 T10 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 8003 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8003 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8004 T61 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@49b9ef36 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@5e4b2b75
[junit4:junit4]   2> 10122 T24 C0 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 10131 T24 C0 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@135d4273 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@778abc7b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10134 T24 C0 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10142 T24 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 783
[junit4:junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@6491cf0b
[junit4:junit4]   2> 10392 T35 C1 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 10395 T35 C1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@20b00ec2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fbb8694; maxCacheMB=0.7099609375 maxMergeSizeMB=0.041015625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10396 T35 C1 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10396 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 5
[junit4:junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@50c3d062
[junit4:junit4]   2> 10405 T50 C2 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 10408 T50 C2 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@67a5fb5a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@79123c5f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10409 T50 C2 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10410 T50 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 6
[junit4:junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@7102be12
[junit4:junit4]   2> 10419 T64 C3 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 10423 T64 C3 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6b0cc9b4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17496e57),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10424 T64 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10425 T64 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 6
[junit4:junit4]   2> 10701 T23 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 266
[junit4:junit4]   2> 10708 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 2
[junit4:junit4]   2> 10716 T24 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 3
[junit4:junit4]   2> 10723 T51 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 3
[junit4:junit4]   2> 10729 T22 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 2
[junit4:junit4]   2> 10735 T65 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 2
[junit4:junit4]   2> 10740 T23 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 1
[junit4:junit4]   2> 10744 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 1
[junit4:junit4]   2> 10750 T24 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 2
[junit4:junit4]   2> 10755 T49 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 2
[junit4:junit4]   2> 10760 T22 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 2
[junit4:junit4]   2> 10767 T63 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 2
[junit4:junit4]   2> 10772 T23 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 1
[junit4:junit4]   2> 10777 T64 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 2
[junit4:junit4]   2> 10820 T24 C0 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 10973 T24 C0 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@135d4273 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@778abc7b),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@135d4273 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@778abc7b),segFN=segments_2,generation=2,filenames=[_0_4.len, _0_2.len, _0_3.len, segments_2, _0.inf, _0.si, _0.pst, _0.fld]
[junit4:junit4]   2> 10974 T24 C0 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 11055 T24 C0 oass.SolrIndexSearcher.<init> Opening Searcher@28bd36fa main
[junit4:junit4]   2> 11055 T24 C0 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 11059 T24 C0 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 11069 T25 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@28bd36fa main{StandardDirectoryReader(segments_2:4 _0(4.1):C7)}
[junit4:junit4]   2> 11071 T24 C0 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 254
[junit4:junit4]   2> 11075 T35 C1 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 11078 T35 C1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@20b00ec2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fbb8694; maxCacheMB=0.7099609375 maxMergeSizeMB=0.041015625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@20b00ec2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@fbb8694; maxCacheMB=0.7099609375 maxMergeSizeMB=0.041015625)),segFN=segments_2,generation=2,filenames=[_0_2.len, _0_3.len, segments_2, _0.inf, _0.si, _0.pst, _0.fld]
[junit4:junit4]   2> 11079 T35 C1 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 11083 T35 C1 oass.SolrIndexSearcher.<init> Opening Searcher@2555e3ab main
[junit4:junit4]   2> 11083 T35 C1 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 11084 T35 C1 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 11088 T37 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2555e3ab main{StandardDirectoryReader(segments_2:4 _0(4.1):C2)}
[junit4:junit4]   2> 11089 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 14
[junit4:junit4]   2> 11093 T51 C2 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 11099 T51 C2 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@67a5fb5a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@79123c5f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@67a5fb5a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@79123c5f),segFN=segments_2,generation=2,filenames=[_0_2.len, _0_3.len, segments_2, _0.inf, _0.si, _0.pst, _0.fld]
[junit4:junit4]   2> 11099 T51 C2 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 11105 T51 C2 oass.SolrIndexSearcher.<init> Opening Searcher@73c28517 main
[junit4:junit4]   2> 11106 T51 C2 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 11106 T51 C2 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 11121 T47 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73c28517 main{StandardDirectoryReader(segments_2:4 _0(4.1):C2)}
[junit4:junit4]   2> 11122 T51 C2 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 29
[junit4:junit4]   2> 11127 T63 C3 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 11129 T63 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6b0cc9b4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17496e57),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6b0cc9b4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@17496e57),segFN=segments_2,generation=2,filenames=[_0_2.len, segments_2, _0.inf, _0.si, _0.pst, _0.fld]
[junit4:junit4]   2> 11130 T63 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 11133 T63 C3 oass.SolrIndexSearcher.<init> Opening Searcher@44ec366a main
[junit4:junit4]   2> 11134 T63 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 11135 T63 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 11139 T61 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@44ec366a main{StandardDirectoryReader(segments_2:4 _0(4.1):C3)}
[junit4:junit4]   2> 11141 T63 C3 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 15
[junit4:junit4]   2> 11990 T23 C0 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&fl=id,+score,+[elevated]&shards.qt=/elevate&q=*:*&distrib=false&qt=/elevate&wt=javabin&rows=500&version=2} hits=7 status=0 QTime=820 
[junit4:junit4]   2> 12036 T49 C2 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351856213370&shard.url=127.0.0.1:57549/solr&fl=id,score&start=0&q=*:*&isShard=true&qt=/elevate&fsv=true} hits=2 status=0 QTime=5 
[junit4:junit4]   2> 12086 T70 C1 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351856213370&shard.url=127.0.0.1:17757/solr|[ff01::213]:33332/solr&fl=id,score&start=0&q=*:*&isShard=true&qt=/elevate&fsv=true} hits=2 status=0 QTime=2 
[junit4:junit4]   2> 12087 T64 C3 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351856213370&shard.url=[ff01::114]:33332/solr|127.0.0.1:56935/solr|[ff01::083]:33332/solr&fl=id,score&start=0&q=*:*&isShard=true&qt=/elevate&fsv=true} hits=3 status=0 QTime=2 
[junit4:junit4]   2> 12113 T51 C2 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=5.0,2.0&distrib=false&wt=javabin&rows=500&version=2&fl=id,+score,+[elevated]&shard.url=127.0.0.1:57549/solr&NOW=1351856213370&q=*:*&qt=/elevate&isShard=true} status=0 QTime=2 
[junit4:junit4]   2> 12113 T70 C1 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=4.0,1.0&distrib=false&wt=javabin&rows=500&version=2&fl=id,+score,+[elevated]&shard.url=127.0.0.1:17757/solr|[ff01::213]:33332/solr&NOW=1351856213370&q=*:*&qt=/elevate&isShard=true} status=0 QTime=1 
[junit4:junit4]   2> 12113 T63 C3 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=3.0,6.0,7.0&distrib=false&wt=javabin&rows=500&version=2&fl=id,+score,+[elevated]&shard.url=[ff01::114]:33332/solr|127.0.0.1:56935/solr|[ff01::083]:33332/solr&NOW=1351856213370&q=*:*&qt=/elevate&isShard=true} status=0 QTime=2 
[junit4:junit4]   2> 12115 T35 C1 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&fl=id,+score,+[elevated]&shards=127.0.0.1:17757/solr|[ff01::213]:33332/solr,127.0.0.1:57549/solr,[ff01::114]:33332/solr|127.0.0.1:56935/solr|[ff01::083]:33332/solr&shards.qt=/elevate&q=*:*&qt=/elevate&wt=javabin&rows=500&version=2} hits=7 status=0 QTime=113 
[junit4:junit4]   2> 12372 T24 C0 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=int_i+desc&shards.qt=/elevate&q=ZZZZ&forceElevation=true&distrib=false&qt=/elevate&wt=javabin&rows=500&version=2} hits=5 status=0 QTime=249 
[junit4:junit4]   2> 12389 T70 C1 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&fl=id&shard.url=127.0.0.1:17757/solr&NOW=1351856213748&start=0&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true&fsv=true} hits=1 status=0 QTime=4 
[junit4:junit4]   2> 12392 T50 C2 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&fl=id&shard.url=127.0.0.1:57549/solr&NOW=1351856213748&start=0&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true&fsv=true} hits=2 status=0 QTime=6 
[junit4:junit4]   2> 12392 T64 C3 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&fl=id&shard.url=[ff01::114]:33332/solr|127.0.0.1:56935/solr&NOW=1351856213748&start=0&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true&fsv=true} hits=2 status=0 QTime=6 
[junit4:junit4]   2> 12398 T70 C1 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=1.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=127.0.0.1:17757/solr&NOW=1351856213748&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=1 
[junit4:junit4]   2> 12399 T49 C2 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=5.0,2.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=127.0.0.1:57549/solr&NOW=1351856213748&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=2 
[junit4:junit4]   2> 12400 T65 C3 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=3.0,6.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=[ff01::114]:33332/solr|127.0.0.1:56935/solr&NOW=1351856213748&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=2 
[junit4:junit4]   2> 12402 T35 C1 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=int_i+desc&shards=127.0.0.1:17757/solr,127.0.0.1:57549/solr,[ff01::114]:33332/solr|127.0.0.1:56935/solr&shards.qt=/elevate&q=ZZZZ&forceElevation=true&qt=/elevate&wt=javabin&rows=500&version=2} hits=5 status=0 QTime=22 
[junit4:junit4]   2> 12423 T22 C0 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=int_i+asc&shards.qt=/elevate&q=solr&forceElevation=true&distrib=false&qt=/elevate&wt=javabin&rows=500&version=2} hits=1 status=0 QTime=17 
[junit4:junit4]   2> 12434 T70 C1 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+asc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351856213796&shard.url=127.0.0.1:17757/solr|[ff01::213]:33332/solr&fl=id&start=0&q=solr&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 12436 T63 C3 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+asc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351856213796&shard.url=[ff01::114]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:56935/solr|[ff01::213]:33332/solr&fl=id&start=0&q=solr&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=1 status=0 QTime=3 
[junit4:junit4]   2> 12437 T51 C2 REQ [collection1] webapp=/solr path=/elevate params={sort=int_i+asc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351856213796&shard.url=127.0.0.1:57549/solr|[ff01::083]:33332/solr&fl=id&start=0&q=solr&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 12445 T65 C3 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=7.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=[ff01::114]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:56935/solr|[ff01::213]:33332/solr&NOW=1351856213796&q=solr&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 12447 T64 C3 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=int_i+asc&shards=127.0.0.1:17757/solr|[ff01::213]:33332/solr,127.0.0.1:57549/solr|[ff01::083]:33332/solr,[ff01::114]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:56935/solr|[ff01::213]:33332/solr&shards.qt=/elevate&q=solr&forceElevation=true&qt=/elevate&wt=javabin&rows=500&version=2} hits=1 status=0 QTime=18 
[junit4:junit4]   2> 12453 T24 C0 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=id+desc&shards.qt=/elevate&q=ZZZZ&forceElevation=true&distrib=false&qt=/elevate&wt=javabin&rows=500&version=2} hits=5 status=0 QTime=3 
[junit4:junit4]   2> 12465 T49 C2 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351856213826&shard.url=[ff01::083]:33332/solr|127.0.0.1:57549/solr&fl=id&start=0&q=ZZZZ&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=2 status=0 QTime=2 
[junit4:junit4]   2> 12466 T35 C1 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351856213826&shard.url=127.0.0.1:17757/solr&fl=id&start=0&q=ZZZZ&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=1 status=0 QTime=2 
[junit4:junit4]   2> 12468 T63 C3 REQ [collection1] webapp=/solr path=/elevate params={sort=id+desc&shards.qt=/elevate&distrib=false&wt=javabin&rows=500&version=2&NOW=1351856213826&shard.url=[ff01::213]:33332/solr|[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:56935/solr&fl=id&start=0&q=ZZZZ&forceElevation=true&isShard=true&qt=/elevate&fsv=true} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 12473 T51 C2 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=5.0,2.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=[ff01::083]:33332/solr|127.0.0.1:57549/solr&NOW=1351856213826&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 12475 T70 C1 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=1.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=127.0.0.1:17757/solr&NOW=1351856213826&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=1 
[junit4:junit4]   2> 12475 T65 C3 REQ [collection1] webapp=/solr path=/elevate params={shards.qt=/elevate&ids=3.0,6.0&distrib=false&wt=javabin&rows=500&version=2&fl=*,+[elevated]&shard.url=[ff01::213]:33332/solr|[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:56935/solr&NOW=1351856213826&q=ZZZZ&forceElevation=true&qt=/elevate&isShard=true} status=0 QTime=1 
[junit4:junit4]   2> 12476 T50 C2 REQ [collection1] webapp=/solr path=/elevate params={fl=*,+[elevated]&sort=id+desc&shards=127.0.0.1:17757/solr,[ff01::083]:33332/solr|127.0.0.1:57549/solr,[ff01::213]:33332/solr|[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:56935/solr&shards.qt=/elevate&q=ZZZZ&forceElevation=true&qt=/elevate&wt=javabin&rows=500&version=2} hits=5 status=0 QTime=18 
[junit4:junit4]   2> 12524 T10 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=2001431298
[junit4:junit4]   2> 12525 T10 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5e4b2b75
[junit4:junit4]   2> 12543 T10 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=7,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 12544 T10 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 12544 T10 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 12544 T10 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 12548 T10 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 12630 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 12732 T10 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 12777 T10 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1592409311
[junit4:junit4]   2> 12778 T10 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6491cf0b
[junit4:junit4]   2> 12787 T10 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 12789 T10 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 12790 T10 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 12791 T10 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 12793 T10 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 12804 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 12905 T10 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 12949 T10 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1085837532
[junit4:junit4]   2> 12950 T10 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@50c3d062
[junit4:junit4]   2> 12956 T10 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=2,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 12957 T10 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 12957 T10 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 12957 T10 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 12958 T10 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 12964 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 13064 T10 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 13109 T10 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=144310161
[junit4:junit4]   2> 13110 T10 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7102be12
[junit4:junit4]   2> 13123 T10 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 13124 T10 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 13125 T10 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 13125 T10 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 13127 T10 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 13134 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 13236 T10 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 13298 T10 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 13321 T10 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 13394 T9 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 16 leaked thread(s).
[junit4:junit4]   2> 33641 T9 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 16 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedQueryElevationComponentTest: 
[junit4:junit4]   2> 	   1) Thread[id=15, name=qtp485367518-15 Acceptor1 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=45, name=qtp810158789-45 Acceptor2 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=34, name=qtp1912354856-34 Acceptor3 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=14, name=qtp485367518-14 Acceptor0 SelectChannelConnector@0.0.0.0:15953, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   5) Thread[id=44, name=qtp810158789-44 Acceptor1 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   6) Thread[id=60, name=qtp1329873750-60 Acceptor3 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   7) Thread[id=19, name=qtp485367518-19 Acceptor2 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   8) Thread[id=58, name=qtp1329873750-58 Acceptor1 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   9) Thread[id=31, name=qtp1912354856-31 Acceptor0 SelectChannelConnector@0.0.0.0:17757, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  10) Thread[id=33, name=qtp1912354856-33 Acceptor2 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  11) Thread[id=59, name=qtp1329873750-59 Acceptor2 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  12) Thread[id=43, name=qtp810158789-43 Acceptor0 SelectChannelConnector@0.0.0.0:57549, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  13) Thread[id=18, name=qtp485367518-18 Acceptor3 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  14) Thread[id=32, name=qtp1912354856-32 Acceptor1 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  15) Thread[id=46, name=qtp810158789-46 Acceptor3 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  16) Thread[id=57, name=qtp1329873750-57 Acceptor0 SelectChannelConnector@0.0.0.0:56935, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 33644 T9 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=15, name=qtp485367518-15 Acceptor1 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   2) Thread[id=45, name=qtp810158789-45 Acceptor2 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   3) Thread[id=34, name=qtp1912354856-34 Acceptor3 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   4) Thread[id=14, name=qtp485367518-14 Acceptor0 SelectChannelConnector@0.0.0.0:15953, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   5) Thread[id=44, name=qtp810158789-44 Acceptor1 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   6) Thread[id=60, name=qtp1329873750-60 Acceptor3 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   7) Thread[id=19, name=qtp485367518-19 Acceptor2 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   8) Thread[id=58, name=qtp1329873750-58 Acceptor1 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	   9) Thread[id=31, name=qtp1912354856-31 Acceptor0 SelectChannelConnector@0.0.0.0:17757, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  10) Thread[id=33, name=qtp1912354856-33 Acceptor2 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  11) Thread[id=59, name=qtp1329873750-59 Acceptor2 SelectChannelConnector@0.0.0.0:56935, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  12) Thread[id=43, name=qtp810158789-43 Acceptor0 SelectChannelConnector@0.0.0.0:57549, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  13) Thread[id=18, name=qtp485367518-18 Acceptor3 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  14) Thread[id=32, name=qtp1912354856-32 Acceptor1 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  15) Thread[id=46, name=qtp810158789-46 Acceptor3 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	  16) Thread[id=57, name=qtp1329873750-57 Acceptor0 SelectChannelConnector@0.0.0.0:56935, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 36834 T9 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=15, name=qtp485367518-15 Acceptor1 SelectChannelConnector@0.0.0.0:15953, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=14, name=qtp485367518-14 Acceptor0 SelectChannelConnector@0.0.0.0:15953, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=34, name=qtp1912354856-34 Acceptor3 SelectChannelConnector@0.0.0.0:17757, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=45, name=qtp810158789-45 Acceptor2 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   5) Thread[id=44, name=qtp810158789-44 Acceptor1 SelectChannelConnector@0.0.0.0:57549, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	

[...truncated too long message...]

     at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=291, name=qtp445567561-291 Acceptor1 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=309, name=qtp131312334-309 Acceptor1 SelectChannelConnector@0.0.0.0:18586, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    9) Thread[id=292, name=qtp445567561-292 Acceptor2 SelectChannelConnector@0.0.0.0:35206, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   10) Thread[id=358, name=qtp1249338498-358 Acceptor2 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=310, name=qtp131312334-310 Acceptor2 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=357, name=qtp1249338498-357 Acceptor1 SelectChannelConnector@0.0.0.0:51985, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=340, name=qtp638227191-340 Acceptor2 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=311, name=qtp131312334-311 Acceptor3 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=308, name=qtp131312334-308 Acceptor0 SelectChannelConnector@0.0.0.0:18586, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=338, name=qtp638227191-338 Acceptor0 SelectChannelConnector@0.0.0.0:21381, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([68B5074705293DC3]:0)
[junit4:junit4] Completed on J0 in 306.39s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 96 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 1 failure, 812 ignored (2 assumptions)

Total time: 23 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 830 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/830/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
64 threads leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandler:     1) Thread[id=43, name=qtp1841925147-43 Acceptor0 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=45, name=qtp1841925147-45 Acceptor2 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=70, name=qtp605441397-70 Acceptor2 SelectChannelConnector@0.0.0.0:56451, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=69, name=qtp605441397-69 Acceptor1 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=29, name=qtp1528972934-29 Acceptor0 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=136, name=qtp646128166-136 Acceptor3 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=230, name=qtp195867017-230 Acceptor3 SelectChannelConnector@0.0.0.0:37028, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=288, name=qtp196245488-288 Acceptor0 SelectChannelConnector@0.0.0.0:51588, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=58, name=qtp609482911-58 Acceptor0 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=186, name=qtp1950405040-186 Acceptor0 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=117, name=qtp1982445652-117 Acceptor3 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=98, name=qtp134827207-98 Acceptor0 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=208, name=qtp283581790-208 Acceptor1 SelectChannelConnector@0.0.0.0:47552, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=149, name=qtp1818331169-149 Acceptor3 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=71, name=qtp605441397-71 Acceptor3 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=32, name=qtp1528972934-32 Acceptor3 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=31, name=qtp1528972934-31 Acceptor2 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=61, name=qtp609482911-61 Acceptor3 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=198, name=qtp2112118876-198 Acceptor1 SelectChannelConnector@0.0.0.0:38463, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=101, name=qtp134827207-101 Acceptor3 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=197, name=qtp2112118876-197 Acceptor0 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=68, name=qtp605441397-68 Acceptor0 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=148, name=qtp1818331169-148 Acceptor2 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=90, name=qtp1784423493-90 Acceptor2 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=133, name=qtp646128166-133 Acceptor0 SelectChannelConnector@0.0.0.0:63816, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=227, name=qtp195867017-227 Acceptor0 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=200, name=qtp2112118876-200 Acceptor3 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=100, name=qtp134827207-100 Acceptor2 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=290, name=qtp196245488-290 Acceptor2 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=147, name=qtp1818331169-147 Acceptor1 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=291, name=qtp196245488-291 Acceptor3 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=134, name=qtp646128166-134 Acceptor1 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=99, name=qtp134827207-99 Acceptor1 SelectChannelConnector@0.0.0.0:20805, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=116, name=qtp1982445652-116 Acceptor2 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=229, name=qtp195867017-229 Acceptor2 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=115, name=qtp1982445652-115 Acceptor1 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   37) Thread[id=210, name=qtp283581790-210 Acceptor3 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   38) Thread[id=273, name=qtp219987657-273 Acceptor1 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   39) Thread[id=274, name=qtp219987657-274 Acceptor2 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   40) Thread[id=243, name=qtp119285778-243 Acceptor2 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   41) Thread[id=88, name=qtp1784423493-88 Acceptor0 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   42) Thread[id=242, name=qtp119285778-242 Acceptor1 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   43) Thread[id=199, name=qtp2112118876-199 Acceptor2 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   44) Thread[id=207, name=qtp283581790-207 Acceptor0 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   45) Thread[id=146, name=qtp1818331169-146 Acceptor0 SelectChannelConnector@0.0.0.0:63174, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   46) Thread[id=114, name=qtp1982445652-114 Acceptor0 SelectChannelConnector@0.0.0.0:58176, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   47) Thread[id=59, name=qtp609482911-59 Acceptor1 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   48) Thread[id=275, name=qtp219987657-275 Acceptor3 SelectChannelConnector@0.0.0.0:17293, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   49) Thread[id=44, name=qtp1841925147-44 Acceptor1 SelectChannelConnector@0.0.0.0:35037, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   50) Thread[id=60, name=qtp609482911-60 Acceptor2 SelectChannelConnector@0.0.0.0:25281, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   51) Thread[id=46, name=qtp1841925147-46 Acceptor3 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   52) Thread[id=209, name=qtp283581790-209 Acceptor2 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   53) Thread[id=135, name=qtp646128166-135 Acceptor2 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   54) Thread[id=289, name=qtp196245488-289 Acceptor1 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   55) Thread[id=244, name=qtp119285778-244 Acceptor3 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   56) Thread[id=189, name=qtp1950405040-189 Acceptor3 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   57) Thread[id=241, name=qtp119285778-241 Acceptor0 SelectChannelConnector@0.0.0.0:18309, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   58) Thread[id=188, name=qtp1950405040-188 Acceptor2 SelectChannelConnector@0.0.0.0:41067, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   59) Thread[id=187, name=qtp1950405040-187 Acceptor1 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   60) Thread[id=89, name=qtp1784423493-89 Acceptor1 SelectChannelConnector@0.0.0.0:38246, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   61) Thread[id=91, name=qtp1784423493-91 Acceptor3 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   62) Thread[id=30, name=qtp1528972934-30 Acceptor1 SelectChannelConnector@0.0.0.0:56479, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   63) Thread[id=228, name=qtp195867017-228 Acceptor1 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   64) Thread[id=272, name=qtp219987657-272 Acceptor0 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 64 threads leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandler: 
   1) Thread[id=43, name=qtp1841925147-43 Acceptor0 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=45, name=qtp1841925147-45 Acceptor2 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=70, name=qtp605441397-70 Acceptor2 SelectChannelConnector@0.0.0.0:56451, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=69, name=qtp605441397-69 Acceptor1 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=29, name=qtp1528972934-29 Acceptor0 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=136, name=qtp646128166-136 Acceptor3 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=230, name=qtp195867017-230 Acceptor3 SelectChannelConnector@0.0.0.0:37028, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=288, name=qtp196245488-288 Acceptor0 SelectChannelConnector@0.0.0.0:51588, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=58, name=qtp609482911-58 Acceptor0 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=186, name=qtp1950405040-186 Acceptor0 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=117, name=qtp1982445652-117 Acceptor3 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=98, name=qtp134827207-98 Acceptor0 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=208, name=qtp283581790-208 Acceptor1 SelectChannelConnector@0.0.0.0:47552, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=149, name=qtp1818331169-149 Acceptor3 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=71, name=qtp605441397-71 Acceptor3 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=32, name=qtp1528972934-32 Acceptor3 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=31, name=qtp1528972934-31 Acceptor2 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=61, name=qtp609482911-61 Acceptor3 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=198, name=qtp2112118876-198 Acceptor1 SelectChannelConnector@0.0.0.0:38463, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=101, name=qtp134827207-101 Acceptor3 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=197, name=qtp2112118876-197 Acceptor0 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=68, name=qtp605441397-68 Acceptor0 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=148, name=qtp1818331169-148 Acceptor2 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=90, name=qtp1784423493-90 Acceptor2 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=133, name=qtp646128166-133 Acceptor0 SelectChannelConnector@0.0.0.0:63816, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=227, name=qtp195867017-227 Acceptor0 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=200, name=qtp2112118876-200 Acceptor3 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=100, name=qtp134827207-100 Acceptor2 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=290, name=qtp196245488-290 Acceptor2 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=147, name=qtp1818331169-147 Acceptor1 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=291, name=qtp196245488-291 Acceptor3 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=134, name=qtp646128166-134 Acceptor1 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=99, name=qtp134827207-99 Acceptor1 SelectChannelConnector@0.0.0.0:20805, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=116, name=qtp1982445652-116 Acceptor2 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=229, name=qtp195867017-229 Acceptor2 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=115, name=qtp1982445652-115 Acceptor1 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  37) Thread[id=210, name=qtp283581790-210 Acceptor3 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  38) Thread[id=273, name=qtp219987657-273 Acceptor1 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  39) Thread[id=274, name=qtp219987657-274 Acceptor2 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  40) Thread[id=243, name=qtp119285778-243 Acceptor2 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  41) Thread[id=88, name=qtp1784423493-88 Acceptor0 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  42) Thread[id=242, name=qtp119285778-242 Acceptor1 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  43) Thread[id=199, name=qtp2112118876-199 Acceptor2 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  44) Thread[id=207, name=qtp283581790-207 Acceptor0 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  45) Thread[id=146, name=qtp1818331169-146 Acceptor0 SelectChannelConnector@0.0.0.0:63174, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  46) Thread[id=114, name=qtp1982445652-114 Acceptor0 SelectChannelConnector@0.0.0.0:58176, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  47) Thread[id=59, name=qtp609482911-59 Acceptor1 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  48) Thread[id=275, name=qtp219987657-275 Acceptor3 SelectChannelConnector@0.0.0.0:17293, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  49) Thread[id=44, name=qtp1841925147-44 Acceptor1 SelectChannelConnector@0.0.0.0:35037, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  50) Thread[id=60, name=qtp609482911-60 Acceptor2 SelectChannelConnector@0.0.0.0:25281, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  51) Thread[id=46, name=qtp1841925147-46 Acceptor3 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  52) Thread[id=209, name=qtp283581790-209 Acceptor2 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  53) Thread[id=135, name=qtp646128166-135 Acceptor2 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  54) Thread[id=289, name=qtp196245488-289 Acceptor1 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  55) Thread[id=244, name=qtp119285778-244 Acceptor3 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  56) Thread[id=189, name=qtp1950405040-189 Acceptor3 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  57) Thread[id=241, name=qtp119285778-241 Acceptor0 SelectChannelConnector@0.0.0.0:18309, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  58) Thread[id=188, name=qtp1950405040-188 Acceptor2 SelectChannelConnector@0.0.0.0:41067, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  59) Thread[id=187, name=qtp1950405040-187 Acceptor1 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  60) Thread[id=89, name=qtp1784423493-89 Acceptor1 SelectChannelConnector@0.0.0.0:38246, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  61) Thread[id=91, name=qtp1784423493-91 Acceptor3 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  62) Thread[id=30, name=qtp1528972934-30 Acceptor1 SelectChannelConnector@0.0.0.0:56479, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  63) Thread[id=228, name=qtp195867017-228 Acceptor1 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  64) Thread[id=272, name=qtp219987657-272 Acceptor0 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([387E652D131B8BB4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=43, name=qtp1841925147-43 Acceptor0 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=45, name=qtp1841925147-45 Acceptor2 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=70, name=qtp605441397-70 Acceptor2 SelectChannelConnector@0.0.0.0:56451, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=69, name=qtp605441397-69 Acceptor1 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=29, name=qtp1528972934-29 Acceptor0 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=136, name=qtp646128166-136 Acceptor3 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=230, name=qtp195867017-230 Acceptor3 SelectChannelConnector@0.0.0.0:37028, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=58, name=qtp609482911-58 Acceptor0 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=288, name=qtp196245488-288 Acceptor0 SelectChannelConnector@0.0.0.0:51588, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=98, name=qtp134827207-98 Acceptor0 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=117, name=qtp1982445652-117 Acceptor3 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=186, name=qtp1950405040-186 Acceptor0 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=208, name=qtp283581790-208 Acceptor1 SelectChannelConnector@0.0.0.0:47552, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=149, name=qtp1818331169-149 Acceptor3 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=71, name=qtp605441397-71 Acceptor3 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=32, name=qtp1528972934-32 Acceptor3 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=31, name=qtp1528972934-31 Acceptor2 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=61, name=qtp609482911-61 Acceptor3 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=198, name=qtp2112118876-198 Acceptor1 SelectChannelConnector@0.0.0.0:38463, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=101, name=qtp134827207-101 Acceptor3 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=197, name=qtp2112118876-197 Acceptor0 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=68, name=qtp605441397-68 Acceptor0 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=148, name=qtp1818331169-148 Acceptor2 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=90, name=qtp1784423493-90 Acceptor2 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=133, name=qtp646128166-133 Acceptor0 SelectChannelConnector@0.0.0.0:63816, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=227, name=qtp195867017-227 Acceptor0 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=200, name=qtp2112118876-200 Acceptor3 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=100, name=qtp134827207-100 Acceptor2 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=290, name=qtp196245488-290 Acceptor2 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=147, name=qtp1818331169-147 Acceptor1 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=291, name=qtp196245488-291 Acceptor3 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=134, name=qtp646128166-134 Acceptor1 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=99, name=qtp134827207-99 Acceptor1 SelectChannelConnector@0.0.0.0:20805, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=116, name=qtp1982445652-116 Acceptor2 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=229, name=qtp195867017-229 Acceptor2 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=115, name=qtp1982445652-115 Acceptor1 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   37) Thread[id=210, name=qtp283581790-210 Acceptor3 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   38) Thread[id=273, name=qtp219987657-273 Acceptor1 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   39) Thread[id=274, name=qtp219987657-274 Acceptor2 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   40) Thread[id=243, name=qtp119285778-243 Acceptor2 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   41) Thread[id=88, name=qtp1784423493-88 Acceptor0 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   42) Thread[id=199, name=qtp2112118876-199 Acceptor2 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   43) Thread[id=242, name=qtp119285778-242 Acceptor1 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   44) Thread[id=207, name=qtp283581790-207 Acceptor0 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   45) Thread[id=59, name=qtp609482911-59 Acceptor1 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   46) Thread[id=114, name=qtp1982445652-114 Acceptor0 SelectChannelConnector@0.0.0.0:58176, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   47) Thread[id=146, name=qtp1818331169-146 Acceptor0 SelectChannelConnector@0.0.0.0:63174, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   48) Thread[id=275, name=qtp219987657-275 Acceptor3 SelectChannelConnector@0.0.0.0:17293, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   49) Thread[id=44, name=qtp1841925147-44 Acceptor1 SelectChannelConnector@0.0.0.0:35037, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   50) Thread[id=60, name=qtp609482911-60 Acceptor2 SelectChannelConnector@0.0.0.0:25281, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   51) Thread[id=46, name=qtp1841925147-46 Acceptor3 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   52) Thread[id=209, name=qtp283581790-209 Acceptor2 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   53) Thread[id=135, name=qtp646128166-135 Acceptor2 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   54) Thread[id=289, name=qtp196245488-289 Acceptor1 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   55) Thread[id=244, name=qtp119285778-244 Acceptor3 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   56) Thread[id=189, name=qtp1950405040-189 Acceptor3 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   57) Thread[id=241, name=qtp119285778-241 Acceptor0 SelectChannelConnector@0.0.0.0:18309, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   58) Thread[id=188, name=qtp1950405040-188 Acceptor2 SelectChannelConnector@0.0.0.0:41067, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   59) Thread[id=187, name=qtp1950405040-187 Acceptor1 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   60) Thread[id=89, name=qtp1784423493-89 Acceptor1 SelectChannelConnector@0.0.0.0:38246, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   61) Thread[id=91, name=qtp1784423493-91 Acceptor3 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   62) Thread[id=30, name=qtp1528972934-30 Acceptor1 SelectChannelConnector@0.0.0.0:56479, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   63) Thread[id=228, name=qtp195867017-228 Acceptor1 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   64) Thread[id=272, name=qtp219987657-272 Acceptor0 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=43, name=qtp1841925147-43 Acceptor0 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=45, name=qtp1841925147-45 Acceptor2 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=70, name=qtp605441397-70 Acceptor2 SelectChannelConnector@0.0.0.0:56451, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=69, name=qtp605441397-69 Acceptor1 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=29, name=qtp1528972934-29 Acceptor0 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=136, name=qtp646128166-136 Acceptor3 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=230, name=qtp195867017-230 Acceptor3 SelectChannelConnector@0.0.0.0:37028, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=58, name=qtp609482911-58 Acceptor0 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=288, name=qtp196245488-288 Acceptor0 SelectChannelConnector@0.0.0.0:51588, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=98, name=qtp134827207-98 Acceptor0 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=117, name=qtp1982445652-117 Acceptor3 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=186, name=qtp1950405040-186 Acceptor0 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=208, name=qtp283581790-208 Acceptor1 SelectChannelConnector@0.0.0.0:47552, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=149, name=qtp1818331169-149 Acceptor3 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=71, name=qtp605441397-71 Acceptor3 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=32, name=qtp1528972934-32 Acceptor3 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=31, name=qtp1528972934-31 Acceptor2 SelectChannelConnector@0.0.0.0:56479, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=61, name=qtp609482911-61 Acceptor3 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=198, name=qtp2112118876-198 Acceptor1 SelectChannelConnector@0.0.0.0:38463, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=101, name=qtp134827207-101 Acceptor3 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=197, name=qtp2112118876-197 Acceptor0 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=68, name=qtp605441397-68 Acceptor0 SelectChannelConnector@0.0.0.0:56451, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=148, name=qtp1818331169-148 Acceptor2 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=90, name=qtp1784423493-90 Acceptor2 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=133, name=qtp646128166-133 Acceptor0 SelectChannelConnector@0.0.0.0:63816, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=227, name=qtp195867017-227 Acceptor0 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=200, name=qtp2112118876-200 Acceptor3 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=100, name=qtp134827207-100 Acceptor2 SelectChannelConnector@0.0.0.0:20805, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=290, name=qtp196245488-290 Acceptor2 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=147, name=qtp1818331169-147 Acceptor1 SelectChannelConnector@0.0.0.0:63174, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=291, name=qtp196245488-291 Acceptor3 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=134, name=qtp646128166-134 Acceptor1 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=99, name=qtp134827207-99 Acceptor1 SelectChannelConnector@0.0.0.0:20805, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=116, name=qtp1982445652-116 Acceptor2 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=229, name=qtp195867017-229 Acceptor2 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=115, name=qtp1982445652-115 Acceptor1 SelectChannelConnector@0.0.0.0:58176, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  37) Thread[id=210, name=qtp283581790-210 Acceptor3 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  38) Thread[id=273, name=qtp219987657-273 Acceptor1 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  39) Thread[id=274, name=qtp219987657-274 Acceptor2 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  40) Thread[id=243, name=qtp119285778-243 Acceptor2 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  41) Thread[id=88, name=qtp1784423493-88 Acceptor0 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  42) Thread[id=199, name=qtp2112118876-199 Acceptor2 SelectChannelConnector@0.0.0.0:38463, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  43) Thread[id=242, name=qtp119285778-242 Acceptor1 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  44) Thread[id=207, name=qtp283581790-207 Acceptor0 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  45) Thread[id=59, name=qtp609482911-59 Acceptor1 SelectChannelConnector@0.0.0.0:25281, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  46) Thread[id=114, name=qtp1982445652-114 Acceptor0 SelectChannelConnector@0.0.0.0:58176, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  47) Thread[id=146, name=qtp1818331169-146 Acceptor0 SelectChannelConnector@0.0.0.0:63174, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  48) Thread[id=275, name=qtp219987657-275 Acceptor3 SelectChannelConnector@0.0.0.0:17293, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  49) Thread[id=44, name=qtp1841925147-44 Acceptor1 SelectChannelConnector@0.0.0.0:35037, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  50) Thread[id=60, name=qtp609482911-60 Acceptor2 SelectChannelConnector@0.0.0.0:25281, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  51) Thread[id=46, name=qtp1841925147-46 Acceptor3 SelectChannelConnector@0.0.0.0:35037, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  52) Thread[id=209, name=qtp283581790-209 Acceptor2 SelectChannelConnector@0.0.0.0:47552, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  53) Thread[id=135, name=qtp646128166-135 Acceptor2 SelectChannelConnector@0.0.0.0:63816, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  54) Thread[id=289, name=qtp196245488-289 Acceptor1 SelectChannelConnector@0.0.0.0:51588, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  55) Thread[id=244, name=qtp119285778-244 Acceptor3 SelectChannelConnector@0.0.0.0:18309, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  56) Thread[id=189, name=qtp1950405040-189 Acceptor3 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  57) Thread[id=241, name=qtp119285778-241 Acceptor0 SelectChannelConnector@0.0.0.0:18309, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  58) Thread[id=188, name=qtp1950405040-188 Acceptor2 SelectChannelConnector@0.0.0.0:41067, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  59) Thread[id=187, name=qtp1950405040-187 Acceptor1 SelectChannelConnector@0.0.0.0:41067, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  60) Thread[id=89, name=qtp1784423493-89 Acceptor1 SelectChannelConnector@0.0.0.0:38246, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  61) Thread[id=91, name=qtp1784423493-91 Acceptor3 SelectChannelConnector@0.0.0.0:38246, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  62) Thread[id=30, name=qtp1528972934-30 Acceptor1 SelectChannelConnector@0.0.0.0:56479, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  63) Thread[id=228, name=qtp195867017-228 Acceptor1 SelectChannelConnector@0.0.0.0:37028, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  64) Thread[id=272, name=qtp219987657-272 Acceptor0 SelectChannelConnector@0.0.0.0:17293, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([387E652D131B8BB4]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedTermsComponentTest

Error Message:
36 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedTermsComponentTest:     1) Thread[id=213, name=qtp1587316788-213 Acceptor3 SelectChannelConnector@0.0.0.0:27324, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=196, name=qtp1079413863-196 Acceptor0 SelectChannelConnector@0.0.0.0:38810, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=123, name=qtp1587572262-123 Acceptor2 SelectChannelConnector@0.0.0.0:31061, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=184, name=qtp1804323940-184 Acceptor0 SelectChannelConnector@0.0.0.0:27475, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=226, name=qtp734411640-226 Acceptor2 SelectChannelConnector@0.0.0.0:51045, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=211, name=qtp1587316788-211 Acceptor1 SelectChannelConnector@0.0.0.0:27324, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=109, name=qtp1024870177-109 Acceptor2 SelectChannelConnector@0.0.0.0:49137, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=138, name=qtp755639182-138 Acceptor1 SelectChannelConnector@0.0.0.0:33502, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=110, name=qtp1024870177-110 Acceptor3 SelectChannelConnector@0.0.0.0:49137, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=212, name=qtp1587316788-212 Acceptor2 SelectChannelConnector@0.0.0.0:27324, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=124, name=qtp1587572262-124 Acceptor3 SelectChannelConnector@0.0.0.0:31061, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=164, name=qtp215134277-164 Acceptor0 SelectChannelConnector@0.0.0.0:11646, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=165, name=qtp215134277-165 Acceptor1 SelectChannelConnector@0.0.0.0:11646, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=150, name=qtp1201114959-150 Acceptor0 SelectChannelConnector@0.0.0.0:23580, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=199, name=qtp1079413863-199 Acceptor3 SelectChannelConnector@0.0.0.0:38810, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=166, name=qtp215134277-166 Acceptor2 SelectChannelConnector@0.0.0.0:11646, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=167, name=qtp215134277-167 Acceptor3 SelectChannelConnector@0.0.0.0:11646, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=108, name=qtp1024870177-108 Acceptor1 SelectChannelConnector@0.0.0.0:49137, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=107, name=qtp1024870177-107 Acceptor0 SelectChannelConnector@0.0.0.0:49137, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=185, name=qtp1804323940-185 Acceptor1 SelectChannelConnector@0.0.0.0:27475, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=151, name=qtp1201114959-151 Acceptor1 SelectChannelConnector@0.0.0.0:23580, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=187, name=qtp1804323940-187 Acceptor3 SelectChannelConnector@0.0.0.0:27475, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=137, name=qtp755639182-137 Acceptor0 SelectChannelConnector@0.0.0.0:33502, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=224, name=qtp734411640-224 Acceptor0 SelectChannelConnector@0.0.0.0:51045, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.

[...truncated too long message...]

junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   27) Thread[id=139, name=qtp755639182-139 Acceptor2 SelectChannelConnector@0.0.0.0:33502, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   28) Thread[id=197, name=qtp1079413863-197 Acceptor1 SelectChannelConnector@0.0.0.0:38810, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   29) Thread[id=225, name=qtp734411640-225 Acceptor1 SelectChannelConnector@0.0.0.0:51045, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   30) Thread[id=152, name=qtp1201114959-152 Acceptor2 SelectChannelConnector@0.0.0.0:23580, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   31) Thread[id=198, name=qtp1079413863-198 Acceptor2 SelectChannelConnector@0.0.0.0:38810, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   32) Thread[id=210, name=qtp1587316788-210 Acceptor0 SelectChannelConnector@0.0.0.0:27324, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   33) Thread[id=227, name=qtp734411640-227 Acceptor3 SelectChannelConnector@0.0.0.0:51045, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   34) Thread[id=121, name=qtp1587572262-121 Acceptor0 SelectChannelConnector@0.0.0.0:31061, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   35) Thread[id=140, name=qtp755639182-140 Acceptor3 SelectChannelConnector@0.0.0.0:33502, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   36) Thread[id=186, name=qtp1804323940-186 Acceptor2 SelectChannelConnector@0.0.0.0:27475, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([387E652D131B8BB4]:0)
[junit4:junit4] Completed on J0 in 36.45s, 1 test, 2 errors <<< FAILURES!

[...truncated 455 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 799 ignored (1 assumption)

Total time: 17 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 829 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/829/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=91, name=qtp106686383-91 Acceptor1 SelectChannelConnector@0.0.0.0:52938, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=93, name=qtp106686383-93 Acceptor3 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=90, name=qtp106686383-90 Acceptor0 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=92, name=qtp106686383-92 Acceptor2 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=182, name=qtp664118822-182 Acceptor1 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=184, name=qtp664118822-184 Acceptor3 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=183, name=qtp664118822-183 Acceptor2 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=181, name=qtp664118822-181 Acceptor0 SelectChannelConnector@0.0.0.0:52944, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest: 
   1) Thread[id=91, name=qtp106686383-91 Acceptor1 SelectChannelConnector@0.0.0.0:52938, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=93, name=qtp106686383-93 Acceptor3 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=90, name=qtp106686383-90 Acceptor0 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=92, name=qtp106686383-92 Acceptor2 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=182, name=qtp664118822-182 Acceptor1 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=184, name=qtp664118822-184 Acceptor3 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=183, name=qtp664118822-183 Acceptor2 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=181, name=qtp664118822-181 Acceptor0 SelectChannelConnector@0.0.0.0:52944, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([429B0813EFF4D0E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=91, name=qtp106686383-91 Acceptor1 SelectChannelConnector@0.0.0.0:52938, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=93, name=qtp106686383-93 Acceptor3 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=90, name=qtp106686383-90 Acceptor0 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=92, name=qtp106686383-92 Acceptor2 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=182, name=qtp664118822-182 Acceptor1 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=184, name=qtp664118822-184 Acceptor3 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=183, name=qtp664118822-183 Acceptor2 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=181, name=qtp664118822-181 Acceptor0 SelectChannelConnector@0.0.0.0:52944, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=91, name=qtp106686383-91 Acceptor1 SelectChannelConnector@0.0.0.0:52938, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=93, name=qtp106686383-93 Acceptor3 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=90, name=qtp106686383-90 Acceptor0 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=92, name=qtp106686383-92 Acceptor2 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=182, name=qtp664118822-182 Acceptor1 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=184, name=qtp664118822-184 Acceptor3 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=183, name=qtp664118822-183 Acceptor2 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=181, name=qtp664118822-181 Acceptor0 SelectChannelConnector@0.0.0.0:52944, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([429B0813EFF4D0E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.SolrCmdDistributorTest

Error Message:
20 threads leaked from SUITE scope at org.apache.solr.update.SolrCmdDistributorTest:     1) Thread[id=238, name=qtp529176285-238 Acceptor3 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=211, name=qtp1185812646-211 Acceptor2 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=197, name=qtp736128313-197 Acceptor2 SelectChannelConnector@0.0.0.0:52964, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=221, name=qtp57538805-221 Acceptor0 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=195, name=qtp736128313-195 Acceptor0 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=224, name=qtp57538805-224 Acceptor3 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=198, name=qtp736128313-198 Acceptor3 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=235, name=qtp529176285-235 Acceptor0 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=249, name=qtp338379097-249 Acceptor0 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=212, name=qtp1185812646-212 Acceptor3 SelectChannelConnector@0.0.0.0:52965, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=210, name=qtp1185812646-210 Acceptor1 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=223, name=qtp57538805-223 Acceptor2 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=237, name=qtp529176285-237 Acceptor2 SelectChannelConnector@0.0.0.0:52967, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=251, name=qtp338379097-251 Acceptor2 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=250, name=qtp338379097-250 Acceptor1 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=196, name=qtp736128313-196 Acceptor1 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=209, name=qtp1185812646-209 Acceptor0 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=236, name=qtp529176285-236 Acceptor1 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=222, name=qtp57538805-222 Acceptor1 SelectChannelConnector@0.0.0.0:52966, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=252, name=qtp338379097-252 Acceptor3 SelectChannelConnector@0.0.0.0:52968, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 20 threads leaked from SUITE scope at org.apache.solr.update.SolrCmdDistributorTest: 
   1) Thread[id=238, name=qtp529176285-238 Acceptor3 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=211, name=qtp1185812646-211 Acceptor2 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=197, name=qtp736128313-197 Acceptor2 SelectChannelConnector@0.0.0.0:52964, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=221, name=qtp57538805-221 Acceptor0 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=195, name=qtp736128313-195 Acceptor0 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=224, name=qtp57538805-224 Acceptor3 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=198, name=qtp736128313-198 Acceptor3 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=235, name=qtp529176285-235 Acceptor0 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=249, name=qtp338379097-249 Acceptor0 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=212, name=qtp1185812646-212 Acceptor3 SelectChannelConnector@0.0.0.0:52965, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=210, name=qtp1185812646-210 Acceptor1 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=223, name=qtp57538805-223 Acceptor2 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=237, name=qtp529176285-237 Acceptor2 SelectChannelConnector@0.0.0.0:52967, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=251, name=qtp338379097-251 Acceptor2 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=250, name=qtp338379097-250 Acceptor1 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=196, name=qtp736128313-196 Acceptor1 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=209, name=qtp1185812646-209 Acceptor0 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=236, name=qtp529176285-236 Acceptor1 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=222, name=qtp57538805-222 Acceptor1 SelectChannelConnector@0.0.0.0:52966, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=252, name=qtp338379097-252 Acceptor3 SelectChannelConnector@0.0.0.0:52968, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([429B0813EFF4D0E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.SolrCmdDistributorTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=197, name=qtp736128313-197 Acceptor2 SelectChannelConnector@0.0.0.0:52964, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=211, name=qtp1185812646-211 Acceptor2 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=238, name=qtp529176285-238 Acceptor3 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=221, name=qtp57538805-221 Acceptor0 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=195, name=qtp736128313-195 Acceptor0 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=198, name=qtp736128313-198 Acceptor3 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=224, name=qtp57538805-224 Acceptor3 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=235, name=qtp529176285-235 Acceptor0 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=249, name=qtp338379097-249 Acceptor0 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=210, name=qtp1185812646-210 Acceptor1 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=212, name=qtp1185812646-212 Acceptor3 SelectChannelConnector@0.0.0.0:52965, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=223, name=qtp57538805-223 Acceptor2 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=237, name=qtp529176285-237 Acceptor2 SelectChannelConnector@0.0.0.0:52967, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=251, name=qtp338379097-251 Acceptor2 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=250, name=qtp338379097-250 Acceptor1 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=196, name=qtp736128313-196 Acceptor1 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=209, name=qtp1185812646-209 Acceptor0 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=236, name=qtp529176285-236 Acceptor1 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=222, name=qtp57538805-222 Acceptor1 SelectChannelConnector@0.0.0.0:52966, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=252, name=qtp338379097-252 Acceptor3 SelectChannelConnector@0.0.0.0:52968, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=197, name=qtp736128313-197 Acceptor2 SelectChannelConnector@0.0.0.0:52964, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=211, name=qtp1185812646-211 Acceptor2 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=238, name=qtp529176285-238 Acceptor3 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=221, name=qtp57538805-221 Acceptor0 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=195, name=qtp736128313-195 Acceptor0 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=198, name=qtp736128313-198 Acceptor3 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=224, name=qtp57538805-224 Acceptor3 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=235, name=qtp529176285-235 Acceptor0 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=249, name=qtp338379097-249 Acceptor0 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=210, name=qtp1185812646-210 Acceptor1 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=212, name=qtp1185812646-212 Acceptor3 SelectChannelConnector@0.0.0.0:52965, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=223, name=qtp57538805-223 Acceptor2 SelectChannelConnector@0.0.0.0:52966, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=237, name=qtp529176285-237 Acceptor2 SelectChannelConnector@0.0.0.0:52967, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=251, name=qtp338379097-251 Acceptor2 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=250, name=qtp338379097-250 Acceptor1 SelectChannelConnector@0.0.0.0:52968, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=196, name=qtp736128313-196 Acceptor1 SelectChannelConnector@0.0.0.0:52964, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=209, name=qtp1185812646-209 Acceptor0 SelectChannelConnector@0.0.0.0:52965, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=236, name=qtp529176285-236 Acceptor1 SelectChannelConnector@0.0.0.0:52967, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=222, name=qtp57538805-222 Acceptor1 SelectChannelConnector@0.0.0.0:52966, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=252, name=qtp338379097-252 Acceptor3 SelectChannelConnector@0.0.0.0:52968, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([429B0813EFF4D0E]:0)




Build Log:
[...truncated 7715 lines...]
[junit4:junit4] Suite: org.apache.solr.update.SolrCmdDistributorTest
[junit4:junit4]   2> 3 T188 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 559 T188 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 804 T188 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:52964
[junit4:junit4]   2> 829 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 830 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 830 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 857 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 858 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 858 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 858 T188 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 859 T188 oasc.CoreContainer.<init> New CoreContainer 147590862
[junit4:junit4]   2> 859 T188 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 859 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 891 T188 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 931 T188 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 932 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 933 T188 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 934 T188 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 962 T188 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 997 T188 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 998 T188 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1005 T188 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1129 T3 oasc.CoreContainer.finalize SEVERE CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!  instance=272123600
[junit4:junit4]   2> 1414 T188 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1422 T188 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1425 T188 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1433 T188 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1437 T188 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1440 T188 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1441 T188 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1442 T188 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/control/data/
[junit4:junit4]   2> 1442 T188 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 1443 T188 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/control/data/index/
[junit4:junit4]   2> 1443 T188 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.SolrCmdDistributorTest-1351802383175/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1444 T188 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/control/data/index forceNew:false
[junit4:junit4]   2> 1447 T188 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1154838 lockFactory=org.apache.lucene.store.NativeFSLockFactory@365a25d3),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1447 T188 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1448 T188 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1448 T188 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1449 T188 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1449 T188 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1454 T188 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1458 T188 oass.SolrIndexSearcher.<init> Opening Searcher@5cd5efe2 main
[junit4:junit4]   2> 1458 T188 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1459 T188 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1459 T188 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1460 T188 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1460 T188 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1461 T188 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1461 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1461 T188 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1461 T188 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1462 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1462 T188 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1462 T188 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1463 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1473 T188 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1473 T203 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5cd5efe2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1473 T188 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 1474 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1524 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1539 T188 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1542 T188 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:52965
[junit4:junit4]   2> 1542 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1543 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1543 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1565 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1565 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1566 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1566 T188 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1566 T188 oasc.CoreContainer.<init> New CoreContainer 85255982
[junit4:junit4]   2> 1567 T188 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1567 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1585 T188 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1599 T188 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1600 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1601 T188 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1601 T188 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1631 T188 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1667 T188 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 1668 T188 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1675 T188 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2068 T188 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2077 T188 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2080 T188 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2089 T188 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2092 T188 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2096 T188 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2097 T188 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2097 T188 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard0/data/
[junit4:junit4]   2> 2098 T188 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 2099 T188 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard0/data/index/
[junit4:junit4]   2> 2099 T188 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2100 T188 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard0/data/index forceNew:false
[junit4:junit4]   2> 2103 T188 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@787527fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@2dd9fc92),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2103 T188 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2104 T188 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2104 T188 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2105 T188 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2105 T188 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2110 T188 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2115 T188 oass.SolrIndexSearcher.<init> Opening Searcher@21a60a44 main
[junit4:junit4]   2> 2115 T188 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 2116 T188 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2116 T188 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2117 T188 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2117 T188 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2117 T188 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2118 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2118 T188 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2118 T188 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2119 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2119 T188 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2119 T188 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2120 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2131 T215 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@21a60a44 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2131 T188 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2132 T188 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2132 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2132 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2140 T188 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2143 T188 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:52966
[junit4:junit4]   2> 2144 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2144 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2144 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2173 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2174 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2174 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2174 T188 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2175 T188 oasc.CoreContainer.<init> New CoreContainer 75221397
[junit4:junit4]   2> 2175 T188 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2176 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2195 T188 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2208 T188 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2209 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2210 T188 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2211 T188 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2240 T188 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2275 T188 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 2276 T188 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2283 T188 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2699 T188 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2707 T188 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2710 T188 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2719 T188 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2723 T188 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2727 T188 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2728 T188 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2729 T188 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard1/data/
[junit4:junit4]   2> 2729 T188 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 2730 T188 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard1/data/index/
[junit4:junit4]   2> 2730 T188 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2731 T188 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard1/data/index forceNew:false
[junit4:junit4]   2> 2734 T188 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@48e0f841 lockFactory=org.apache.lucene.store.NativeFSLockFactory@33f487e7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2734 T188 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2735 T188 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2736 T188 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2736 T188 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2737 T188 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2742 T188 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2747 T188 oass.SolrIndexSearcher.<init> Opening Searcher@5189112b main
[junit4:junit4]   2> 2748 T188 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 2749 T188 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2749 T188 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2749 T188 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2750 T188 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2750 T188 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2750 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2751 T188 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2751 T188 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2751 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2752 T188 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2752 T188 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2752 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2764 T229 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5189112b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2764 T188 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2765 T188 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2765 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2765 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2774 T188 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2776 T188 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:52967
[junit4:junit4]   2> 2777 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2778 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2778 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2811 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2812 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2812 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2812 T188 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2813 T188 oasc.CoreContainer.<init> New CoreContainer 6844336
[junit4:junit4]   2> 2814 T188 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2814 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2834 T188 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2847 T188 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2848 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2849 T188 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2849 T188 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2882 T188 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2922 T188 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 2923 T188 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2932 T188 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 3414 T188 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 3423 T188 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 3426 T188 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 3435 T188 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3439 T188 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3442 T188 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3443 T188 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3444 T188 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard2/data/
[junit4:junit4]   2> 3444 T188 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 3445 T188 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard2/data/index/
[junit4:junit4]   2> 3446 T188 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 3446 T188 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard2/data/index forceNew:false
[junit4:junit4]   2> 3449 T188 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6cbe8dd0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@25b27332),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3450 T188 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3451 T188 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 3451 T188 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 3452 T188 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 3452 T188 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 3457 T188 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 3462 T188 oass.SolrIndexSearcher.<init> Opening Searcher@527b1b10 main
[junit4:junit4]   2> 3462 T188 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3463 T188 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 3464 T188 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 3464 T188 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 3464 T188 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3465 T188 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 3465 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3465 T188 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3466 T188 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3466 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3466 T188 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3466 T188 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3467 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 3480 T243 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@527b1b10 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 3480 T188 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 3481 T188 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3481 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3481 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3490 T188 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 3492 T188 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:52968
[junit4:junit4]   2> 3493 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3493 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 3494 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3516 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 3517 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3517 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 3517 T188 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 3518 T188 oasc.CoreContainer.<init> New CoreContainer 2089480659
[junit4:junit4]   2> 3518 T188 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3519 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3536 T188 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3549 T188 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 3550 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 3551 T188 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 3552 T188 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 3583 T188 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 3617 T188 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 3618 T188 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 3627 T188 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 4048 T188 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 4058 T188 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 4061 T188 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 4069 T188 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4073 T188 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4076 T188 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4078 T188 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4078 T188 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard3/data/
[junit4:junit4]   2> 4079 T188 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 4080 T188 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard3/data/index/
[junit4:junit4]   2> 4080 T188 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard3/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 4081 T188 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.update.SolrCmdDistributorTest-1351802383175/shard3/data/index forceNew:false
[junit4:junit4]   2> 4084 T188 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@23e77920 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3d34dcb),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4084 T188 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4085 T188 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 4086 T188 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 4086 T188 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 4086 T188 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 4093 T188 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 4098 T188 oass.SolrIndexSearcher.<init> Opening Searcher@21dd3180 main
[junit4:junit4]   2> 4098 T188 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4099 T188 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 4100 T188 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 4100 T188 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 4100 T188 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4101 T188 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 4101 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4101 T188 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4102 T188 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4102 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4102 T188 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4103 T188 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4103 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 4115 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@21dd3180 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 4115 T188 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 4116 T188 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 4116 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 4117 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C14 name=collection1 org.apache.solr.core.SolrCore@61e2cf66
[junit4:junit4]   2> 5422 T201 C14 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1154838 lockFactory=org.apache.lucene.store.NativeFSLockFactory@365a25d3),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5423 T201 C14 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5424 T201 C14 oass.SolrIndexSearcher.<init> Opening Searcher@4d586d6a realtime
[junit4:junit4]   2> 5425 T201 C14 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5426 T201 C14 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1417467541418147840)} 0 38
[junit4:junit4]   2> ASYNC  NEW_CORE C15 name=collection1 org.apache.solr.core.SolrCore@61da1de
[junit4:junit4]   2> 5602 T214 C15 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@787527fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@2dd9fc92),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5603 T214 C15 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5604 T214 C15 oass.SolrIndexSearcher.<init> Opening Searcher@646f24ee realtime
[junit4:junit4]   2> 5605 T214 C15 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5605 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1417467541606891520)} 0 6
[junit4:junit4]   2> ASYNC  NEW_CORE C16 name=collection1 org.apache.solr.core.SolrCore@7340be40
[junit4:junit4]   2> 5616 T228 C16 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@48e0f841 lockFactory=org.apache.lucene.store.NativeFSLockFactory@33f487e7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5617 T228 C16 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5619 T228 C16 oass.SolrIndexSearcher.<init> Opening Searcher@618c79b9 realtime
[junit4:junit4]   2> 5619 T228 C16 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5620 T228 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1417467541621571584)} 0 7
[junit4:junit4]   2> ASYNC  NEW_CORE C17 name=collection1 org.apache.solr.core.SolrCore@4ecb8779
[junit4:junit4]   2> 5630 T242 C17 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6cbe8dd0 lockFactory=org.apache.lucene.store.NativeFSLockFactory@25b27332),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5631 T242 C17 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5633 T242 C17 oass.SolrIndexSearcher.<init> Opening Searcher@1f9c621d realtime
[junit4:junit4]   2> 5633 T242 C17 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5634 T242 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1417467541637300224)} 0 6
[junit4:junit4]   2> ASYNC  NEW_CORE C18 name=collection1 org.apache.solr.core.SolrCore@c7b682f
[junit4:junit4]   2> 5645 T253 C18 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@23e77920 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3d34dcb),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5645 T253 C18 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5648 T253 C18 oass.SolrIndexSearcher.<init> Opening Searcher@63162cdf realtime
[junit4:junit4]   2> 5648 T253 C18 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5649 T253 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1417467541651980288)} 0 8
[junit4:junit4]   2> 5665 T188 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16
[junit4:junit4]   2> 5744 T200 C14 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1 (1417467541755789312)]} 0 3
[junit4:junit4]   2> 5757 T188 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:52964/solr/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 5787 T202 C14 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5816 T202 C14 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1154838 lockFactory=org.apache.lucene.store.NativeFSLockFactory@365a25d3),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1154838 lockFactory=org.apache.lucene.store.NativeFSLockFactory@365a25d3),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Memory_0.ram, _0_SimpleText_0.pst, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Memory_1.ram, _0.fdt]
[junit4:junit4]   2> 5817 T202 C14 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 5830 T202 C14 oass.SolrIndexSearcher.<init> Opening Searcher@401623f main
[junit4:junit4]   2> 5830 T202 C14 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5831 T203 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@401623f main{StandardDirectoryReader(segments_2:3:nrt _0(4.1):C1)}
[junit4:junit4]   2> 5831 T202 C14 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5832 T202 C14 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 45
[junit4:junit4]   2> 5923 T201 C14 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=1 status=0 QTime=53 
[junit4:junit4]   2> 5930 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&commit_end_point=true&version=2} {add=[2 (1417467541950824448), 3 (1417467541952921600), 4 (1417467541952921601)]} 0 3
[junit4:junit4]   2> 5935 T202 C14 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&commit_end_point=true&version=2} {add=[2 (1417467541952921600), 3 (1417467541958164480)]} 0 7
[junit4:junit4]   2> 5936 T188 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:52964/solr/, StdNode: http://127.0.0.1:52965/solr/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 5940 T214 C15 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5940 T200 C14 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5943 T214 C15 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@787527fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@2dd9fc92),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@787527fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@2dd9fc92),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Memory_0.ram, _0_SimpleText_0.pst, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Memory_1.ram, _0.fdt]
[junit4:junit4]   2> 5943 T214 C15 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 5945 T214 C15 oass.SolrIndexSearcher.<init> Opening Searcher@4988e33f main
[junit4:junit4]   2> 5945 T214 C15 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5945 T214 C15 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5945 T215 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4988e33f main{StandardDirectoryReader(segments_2:3:nrt _0(4.1):C3)}
[junit4:junit4]   2> 5946 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 6
[junit4:junit4]   2> 5950 T200 C14 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1154838 lockFactory=org.apache.lucene.store.NativeFSLockFactory@365a25d3),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Memory_0.ram, _0_SimpleText_0.pst, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Memory_1.ram, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1154838 lockFactory=org.apache.lucene.store.NativeFSLockFactory@365a25d3),segFN=segments_3,generation=3,filenames=[_1_nrm.cfs, _0.fnm, _1_Memory_1.ram, _0_Memory_0.ram, _0_nrm.cfs, _1.fnm, _0_Memory_1.ram, _1_nrm.cfe, _1.fdx, _1_Memory_0.ram, _1.fdt, _1.si, _0_nrm.cfe, _0_SimpleText_0.pst, _0.fdx, _0.si, segments_3, _0.fdt, _1_SimpleText_0.pst]
[junit4:junit4]   2> 5950 T200 C14 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 5952 T200 C14 oass.SolrIndexSearcher.<init> Opening Searcher@6cdb177d main
[junit4:junit4]   2> 5952 T200 C14 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5953 T203 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6cdb177d main{StandardDirectoryReader(segments_3:5:nrt _0(4.1):C1 _1(4.1):C2)}
[junit4:junit4]   2> 5953 T200 C14 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5953 T200 C14 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 13
[junit4:junit4]   2> 5957 T201 C14 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=3 status=0 QTime=1 
[junit4:junit4]   2> 5967 T214 C15 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=3 status=0 QTime=0 
[junit4:junit4]   2> 5980 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&commit_end_point=true&version=2} {delete=[2 (-1417467542004301824)]} 0 2
[junit4:junit4]   2> 5980 T202 C14 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&commit_end_point=true&version=2} {delete=[2 (-1417467542004301824)]} 0 2
[junit4:junit4]   2> 5981 T188 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:52964/solr/, StdNode: http://127.0.0.1:52965/solr/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 5985 T214 C15 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5986 T200 C14 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5987 T214 C15 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@787527fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@2dd9fc92),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Memory_0.ram, _0_SimpleText_0.pst, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Memory_1.ram, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@787527fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@2dd9fc92),segFN=segments_3,generation=3,filenames=[_0.fnm, _0_Memory_0.ram, _0_SimpleText_0.pst, _0_nrm.cfe, _0_1.del, _0.fdx, _0_nrm.cfs, _0.si, _0_Memory_1.ram, _0.fdt, segments_3]
[junit4:junit4]   2> 5987 T200 C14 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1154838 lockFactory=org.apache.lucene.store.NativeFSLockFactory@365a25d3),segFN=segments_3,generation=3,filenames=[_1_nrm.cfs, _0.fnm, _1_Memory_1.ram, _0_Memory_0.ram, _0_nrm.cfs, _1.fnm, _0_Memory_1.ram, _1_nrm.cfe, _1.fdx, _1_Memory_0.ram, _1.fdt, _1.si, _0_nrm.cfe, _0_SimpleText_0.pst, _0.fdx, _0.si, segments_3, _0.fdt, _1_SimpleText_0.pst]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1154838 lockFactory=org.apache.lucene.store.NativeFSLockFactory@365a25d3),segFN=segments_4,generation=4,filenames=[_1_nrm.cfs, _0.fnm, _1_Memory_1.ram, _0_Memory_0.ram, _0_nrm.cfs, _1_1.del, _1.fnm, _0_Memory_1.ram, _1_nrm.cfe, _1.fdx, _1_Memory_0.ram, _1.fdt, _1.si, _0_nrm.cfe, _0_SimpleText_0.pst, _0.fdx, _0.si, _0.fdt, _1_SimpleText_0.pst, segments_4]
[junit4:junit4]   2> 5987 T214 C15 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 5988 T200 C14 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 5988 T200 C14 oass.SolrIndexSearcher.<init> Opening Searcher@4a08e396 main
[junit4:junit4]   2> 5988 T214 C15 oass.SolrIndexSearcher.<init> Opening Searcher@1688e9ff main
[junit4:junit4]   2> 5989 T200 C14 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5989 T214 C15 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5989 T203 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a08e396 main{StandardDirectoryReader(segments_4:6:nrt _0(4.1):C1 _1(4.1):C2/1)}
[junit4:junit4]   2> 5989 T200 C14 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5989 T214 C15 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5989 T215 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1688e9ff main{StandardDirectoryReader(segments_3:4:nrt _0(4.1):C3/1)}
[junit4:junit4]   2> 5990 T200 C14 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 4
[junit4:junit4]   2> 5990 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 5
[junit4:junit4]   2> 5994 T201 C14 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=2 status=0 QTime=0 
[junit4:junit4]   2> 5999 T214 C15 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=2 status=0 QTime=0 
[junit4:junit4]   2> 6004 T214 C15 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6011 T214 C15 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@787527fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@2dd9fc92),segFN=segments_3,generation=3,filenames=[_0.fnm, _0_Memory_0.ram, _0_SimpleText_0.pst, _0_nrm.cfe, _0_1.del, _0.fdx, _0_nrm.cfs, _0.si, _0_Memory_1.ram, _0.fdt, segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@787527fc lockFactory=org.apache.lucene.store.NativeFSLockFactory@2dd9fc92),segFN=segments_4,generation=4,filenames=[_1.fdx, _1_nrm.cfs, _1_Memory_0.ram, _1.fdt, _1_Memory_1.ram, _1.si, _1.fnm, _1_SimpleText_0.pst, _1_nrm.cfe, segments_4]
[junit4:junit4]   2> 6011 T214 C15 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 6014 T214 C15 oass.SolrIndexSearcher.<init> Opening Searcher@527dea56 main
[junit4:junit4]   2> 6014 T214 C15 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6015 T215 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@527dea56 main{StandardDirectoryReader(segments_4:6:nrt _1(4.1):C2)}
[junit4:junit4]   2> 6015 T214 C15 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6016 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={optimize=true&maxSegments=1&waitSearcher=true&wt=javabin&version=2} {optimize=} 0 12
[junit4:junit4]   2> 6020 T227 C16 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6023 T227 C16 oass.SolrIndexSearcher.<init> Opening Searcher@119bc7c main
[junit4:junit4]   2> 6023 T227 C16 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6024 T229 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@119bc7c main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 6024 T227 C16 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6024 T227 C16 UPDATE [collection1] webapp=/solr path=/update params={optimize=true&maxSegments=1&waitSearcher=true&wt=javabin&version=2} {optimize=} 0 4
[junit4:junit4]   2> 6029 T240 C17 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6031 T240 C17 oass.SolrIndexSearcher.<init> Opening Searcher@42d4a1cc main
[junit4:junit4]   2> 6031 T240 C17 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6032 T243 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@42d4a1cc main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 6032 T240 C17 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6032 T240 C17 UPDATE [collection1] webapp=/solr path=/update params={optimize=true&maxSegments=1&waitSearcher=true&wt=javabin&version=2} {optimize=} 0 3
[junit4:junit4]   2> 6036 T256 C18 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6037 T256 C18 oass.SolrIndexSearcher.<init> Opening Searcher@5164a171 main
[junit4:junit4]   2> 6038 T256 C18 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6038 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5164a171 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 6038 T256 C18 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6038 T256 C18 UPDATE [collection1] webapp=/solr path=/update params={optimize=true&maxSegments=1&waitSearcher=true&wt=javabin&version=2} {optimize=} 0 2
[junit4:junit4]   2> 6233 T255 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6 (1417467542072459264), 7 (1417467542266445824), 9 (1417467542267494400), 10 (1417467542267494401), 13 (1417467542268542976), 15 (1417467542268542977), 17 (1417467542269591552), 20 (1417467542270640128), 21 (1417467542270640129), 22 (1417467542271688704)]} 0 190
[junit4:junit4]   2> 6234 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7 (1417467542265397248), 17 (1417467542267494400), 18 (1417467542267494401), 25 (1417467542268542976), 26 (1417467542269591552), 27 (1417467542269591553), 31 (1417467542270640128), 33 (1417467542270640129), 34 (1417467542271688704), 35 (1417467542271688705)]} 0 7
[junit4:junit4]   2> 6252 T228 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23 (1417467542074556416), 24 (1417467542274834432), 26 (1417467542281125888), 31 (1417467542283223040), 33 (1417467542284271616), 34 (1417467542285320192), 35 (1417467542286368768), 37 (1417467542286368769), 39 (1417467542288465920), 44 (1417467542290563072)]} 0 207
[junit4:junit4]   2> 6253 T242 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5 (1417467542074556416), 9 (1417467542282174464), 10 (1417467542282174465), 14 (1417467542284271616), 15 (1417467542285320192), 17 (1417467542287417344), 19 (1417467542288465921), 21 (1417467542289514497), 32 (1417467542290563072), 36 (1417467542291611648)]} 0 208
[junit4:junit4]   2> 6431 T253 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[44 (1417467542284271617), 46 (1417467542286368768), 47 (1417467542289514496), 48 (1417467542289514497), 50 (1417467542290563072), 52 (1417467542290563073), 55 (1417467542291611648), 57 (1417467542292660224), 59 (1417467542292660225), 62 (1417467542310486016)]} 0 186
[junit4:junit4]   2> 6518 T281 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[93 (1417467542485598208), 100 (1417467542546415617), 101 (1417467542547464192), 103 (1417467542548512768), 105 (1417467542549561344), 106 (1417467542549561345), 108 (1417467542550609920), 109 (1417467542551658496), 113 (1417467542551658497), 115 (1417467542552707072)]} 0 81
[junit4:junit4]   2> 6501 T241 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[39 (1417467542281125888), 40 (1417467542283223040), 42 (1417467542284271617), 43 (1417467542285320193), 46 (1417467542288465920), 47 (1417467542289514496), 48 (1417467542311534592), 49 (1417467542478258176), 50 (1417467542479306753), 52 (1417467542483501057)]} 0 261
[junit4:junit4]   2> 6500 T228 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[66 (1417467542478258176), 67 (1417467542479306752), 70 (1417467542483501056), 71 (1417467542546415617), 72 (1417467542547464192), 73 (1417467542547464194), 74 (1417467542548512769), 83 (1417467542549561344), 84 (1417467542549561346), 85 (1417467542550609920)]} 0 71
[junit4:junit4]   2> 6499 T227 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5 (1417467542274834433), 6 (1417467542282174464), 7 (1417467542477209600), 9 (1417467542478258177), 11 (1417467542479306753), 12 (1417467542484549632), 13 (1417467542546415616), 16 (1417467542547464193), 19 (1417467542548512768), 20 (1417467542549561345)]} 0 263
[junit4:junit4]   2> 6499 T240 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[54 (1417467542285320194), 57 (1417467542479306752), 59 (1417467542483501056), 65 (1417467542546415616), 67 (1417467542546415617), 69 (1417467542547464192), 71 (1417467542548512768), 73 (1417467542548512769), 75 (1417467542549561344), 76 (1417467542550609920)]} 0 253
[junit4:junit4]   2> 6496 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[36 (1417467542282174464), 38 (1417467542283223040), 40 (1417467542284271616), 43 (1417467542285320192), 44 (1417467542286368769), 46 (1417467542310486016), 47 (1417467542478258176), 51 (1417467542479306752), 54 (1417467542483501056), 55 (1417467542546415616)]} 0 253
[junit4:junit4]   2> 6495 T226 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[45 (1417467542287417344), 46 (1417467542289514496), 50 (1417467542291611648), 53 (1417467542292660224), 56 (1417467542293708800), 57 (1417467542294757376), 59 (1417467542295805952), 60 (1417467542310486016), 61 (1417467542478258178), 62 (1417467542480355328)]} 0 247
[junit4:junit4]   2> 6436 T256 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24 (1417467542265397248), 26 (1417467542277980160), 28 (1417467542281125888), 31 (1417467542282174464), 33 (1417467542283223040), 35 (1417467542284271616), 39 (1417467542285320192), 40 (1417467542286368769), 41 (1417467542478258178), 43 (1417467542479306753)]} 0 208
[junit4:junit4]   2> 6625 T283 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[113 (1417467542570532864), 114 (1417467542571581440), 116 (1417467542574727168), 117 (1417467542575775744), 119 (1417467542576824320), 120 (1417467542577872896), 121 (1417467542578921472), 123 (1417467542579970048), 124 (1417467542581018624), 128 (1417467542676439040)]} 0 105
[junit4:junit4]   2> 6620 T296 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[85 (1417467542575775746), 88 (1417467542576824320), 89 (1417467542577872896), 90 (1417467542578921472), 94 (1417467542578921473), 95 (1417467542579970048), 97 (1417467542581018624), 98 (1417467542582067200), 99 (1417467542583115776), 100 (1417467542676439040)]} 0 97
[junit4:junit4]   2> 6533 T285 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[138 (1417467542575775746), 141 (1417467542578921472), 143 (1417467542579970048), 144 (1417467542581018624), 146 (1417467542582067200), 148 (1417467542583115776), 152 (1417467542583115777), 153 (1417467542584164352), 154 (1417467542585212928), 156 (1417467542585212929)]} 0 10
[junit4:junit4]   2> 6627 T271 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[56 (1417467542286368768), 58 (1417467542477209600), 61 (1417467542478258177), 63 (1417467542544318464), 64 (1417467542575775748), 68 (1417467542576824320), 70 (1417467542679584768), 71 (1417467542682730497), 74 (1417467542682730498), 77 (1417467542684827648)]} 0 380
[junit4:junit4]   2> 6520 T255 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[63 (1417467542478258177), 67 (1417467542479306752), 68 (1417467542483501056), 70 (1417467542484549632), 72 (1417467542548512768), 76 (1417467542549561344), 79 (1417467542550609920), 80 (1417467542552707072), 82 (1417467542569484288), 84 (1417467542571581440)]} 0 90
[junit4:junit4]   2> 6519 T279 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[101 (1417467542478258176), 102 (1417467542479306754), 103 (1417467542485598208), 109 (1417467542485598209), 112 (1417467542546415616), 116 (1417467542547464192), 117 (1417467542548512769), 118 (1417467542549561345), 120 (1417467542551658496), 121 (1417467542552707073)]} 0 90
[junit4:junit4]   2> 6626 T241 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[100 (1417467542574727168), 102 (1417467542575775744), 103 (1417467542577872896), 104 (1417467542578921472), 105 (1417467542579970048), 107 (1417467542581018624), 108 (1417467542582067200), 109 (1417467542583115776), 110 (1417467542676439040), 111 (1417467542680633344)]} 0 104
[junit4:junit4]   2> 6636 T240 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[80 (1417467542681681921), 84 (1417467542684827648), 86 (1417467542685876224), 88 (1417467542686924802), 91 (1417467542687973377), 93 (1417467542689021953), 94 (1417467542690070529), 96 (1417467542692167680), 97 (1417467542693216257), 99 (1417467542693216258)]} 0 14
[junit4:junit4]   2> 6638 T297 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[140 (1417467542575775744), 141 (1417467542575775747), 142 (1417467542679584768), 145 (1417467542686924800), 146 (1417467542687973376), 147 (1417467542689021952), 148 (1417467542692167680), 150 (1417467542693216256), 151 (1417467542694264832), 154 (1417467542695313409)]} 0 116
[junit4:junit4]   2> 6641 T286 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[79 (1417467542574727168), 81 (1417467542575775745), 84 (1417467542677487616), 85 (1417467542682730496), 86 (1417467542684827649), 87 (1417467542684827651), 89 (1417467542686924800), 90 (1417467542689021952), 91 (1417467542691119104), 92 (1417467542694264832)]} 0 123
[junit4:junit4]   2> 6645 T295 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[116 (1417467542575775744), 119 (1417467542575775747), 120 (1417467542680633344), 121 (1417467542684827650), 126 (1417467542687973376), 127 (1417467542690070528), 129 (1417467542692167680), 130 (1417467542693216256), 134 (1417467542699507712), 135 (1417467542701604866)]} 0 123
[junit4:junit4]   2> 6646 T282 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[129 (1417467542570532865), 131 (1417467542572630016), 134 (1417467542679584769), 135 (1417467542689021952), 137 (1417467542690070528), 138 (1417467542693216256), 139 (1417467542695313408), 140 (1417467542696361984), 141 (1417467542700556293), 142 (1417467542702653441)]} 0 128
[junit4:junit4]   2> 6648 T242 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[112 (1417467542679584768), 113 (1417467542686924800), 117 (1417467542687973376), 120 (1417467542689021954), 121 (1417467542691119105), 123 (1417467542693216256), 124 (1417467542700556288), 126 (1417467542701604865), 130 (1417467542703702016), 133 (1417467542704750592)]} 0 26
[junit4:junit4]   2> 6650 T226 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[162 (1417467542695313409), 165 (1417467542696361987), 168 (1417467542700556288), 177 (1417467542700556292), 179 (1417467542701604864), 181 (1417467542703702017), 182 (1417467542704750593), 185 (1417467542705799169), 186 (1417467542706847744), 188 (1417467542707896320)]} 0 21
[junit4:junit4]   2> 6652 T240 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[234 (1417467542701604864), 238 (1417467542702653440), 239 (1417467542703702018), 242 (1417467542705799168), 244 (1417467542705799169), 248 (1417467542706847745), 252 (1417467542707896322), 255 (1417467542708944896), 256 (1417467542708944898), 258 (1417467542709993473)]} 0 9
[junit4:junit4]   2> 6653 T297 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[192 (1417467542701604864), 193 (1417467542702653440), 194 (1417467542703702016), 196 (1417467542704750592), 197 (1417467542705799168), 201 (1417467542705799169), 202 (1417467542708944896), 204 (1417467542709993472), 206 (1417467542709993473), 209 (1417467542711042048)]} 0 10
[junit4:junit4]   2> 6656 T309 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[207 (1417467542702653440), 208 (1417467542703702016), 210 (1417467542705799168), 212 (1417467542707896321), 213 (1417467542708944896), 215 (1417467542709993472), 216 (1417467542711042048), 218 (1417467542712090624), 220 (1417467542713139200), 222 (1417467542714187776)]} 0 13
[junit4:junit4]   2> 6696 T256 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[211 (1417467542695313408), 212 (1417467542696361984), 213 (1417467542696361985), 214 (1417467542699507712), 217 (1417467542750887936), 218 (1417467542751936513), 219 (1417467542752985088), 221 (1417467542754033664), 222 (1417467542755082241), 223 (1417467542756130817)]} 0 59
[junit4:junit4]   2> 6731 T228 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[189 (1417467542750887936), 191 (1417467542763470848), 193 (1417467542763470851), 194 (1417467542764519427), 198 (1417467542766616579), 200 (1417467542770810880), 201 (1417467542771859457), 203 (1417467542772908033), 204 (1417467542778150912), 205 (1417467542779199490)]} 0 89
[junit4:junit4]   2> 6732 T271 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[230 (1417467542768713729), 233 (1417467542773956608), 236 (1417467542776053760), 237 (1417467542777102337), 238 (1417467542778150912), 241 (1417467542779199488), 242 (1417467542780248067), 244 (1417467542781296641), 245 (1417467542782345217), 246 (1417467542791782404)]} 0 25
[junit4:junit4]   2> 6731 T311 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[260 (1417467542771859456), 261 (1417467542772908032), 262 (1417467542772908033), 264 (1417467542783393792), 268 (1417467542784442370), 269 (1417467542785490946), 270 (1417467542787588098), 272 (1417467542789685248), 275 (1417467542790733824), 277 (1417467542792830976)]} 0 21
[junit4:junit4]   2> 6729 T320 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[318 (1417467542764519424), 319 (1417467542781296643), 320 (1417467542784442368), 321 (1417467542785490944), 322 (1417467542786539521), 323 (1417467542787588097), 324 (1417467542788636672), 325 (1417467542789685249), 326 (1417467542790733824), 327 (1417467542791782400)]} 0 26
[junit4:junit4]   2> 6733 T300 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[174 (1417467542696361984), 175 (1417467542701604865), 176 (1417467542761373697), 177 (1417467542783393795), 178 (1417467542787588096), 179 (1417467542788636673), 181 (1417467542790733825), 182 (1417467542791782401), 185 (1417467542793879554), 187 (1417467542794928128)]} 0 96
[junit4:junit4]   2> 6729 T286 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[297 (1417467542766616576), 301 (1417467542768713730), 306 (1417467542770810880), 307 (1417467542771859459), 308 (1417467542775005184), 309 (1417467542778150913), 310 (1417467542780248065), 311 (1417467542782345218), 314 (1417467542787588098), 315 (1417467542789685248)]} 0 25
[junit4:junit4]   2> 6728 T315 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[223 (1417467542707896321), 224 (1417467542708944897), 226 (1417467542763470848), 227 (1417467542763470850), 228 (1417467542766616576), 229 (1417467542767665152), 230 (1417467542768713729), 231 (1417467542785490945), 232 (1417467542787588096), 233 (1417467542789685249)]} 0 79
[junit4:junit4]   2> 6728 T338 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[380 (1417467542773956608), 381 (1417467542776053761), 383 (1417467542777102338), 385 (1417467542779199488), 387 (1417467542780248066), 388 (1417467542781296642), 391 (1417467542783393792), 392 (1417467542784442369), 393 (1417467542787588096), 394 (1417467542788636674)]} 0 16
[junit4:junit4]   2> 6739 T302 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[160 (1417467542690070528), 161 (1417467542691119104), 162 (1417467542717333506), 163 (1417467542764519424), 168 (1417467542785490944), 169 (1417467542786539520), 170 (1417467542795976704), 171 (1417467542798073856), 173 (1417467542800171009), 178 (1417467542801219584)]} 0 108
[junit4:junit4]   2> 6727 T241 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[204 (1417467542696361984), 205 (1417467542702653442), 208 (1417467542706847744), 209 (1417467542707896323), 211 (1417467542709993472), 213 (1417467542711042048), 214 (1417467542712090624), 217 (1417467542762422273), 218 (1417467542784442368), 219 (1417467542787588097)]} 0 89
[junit4:junit4]   2> 6725 T285 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[190 (1417467542751936512), 191 (1417467542754033664), 192 (1417467542755082240), 194 (1417467542757179392), 196 (1417467542758227968), 198 (1417467542759276544), 199 (1417467542771859457), 201 (1417467542773956611), 202 (1417467542780248064), 203 (1417467542781296642)]} 0 33
[junit4:junit4]   2> 6719 T255 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[307 (1417467542762422272), 308 (1417467542763470848), 309 (1417467542766616577), 311 (1417467542768713728), 320 (1417467542769762305), 322 (1417467542771859458), 323 (1417467542772908033), 325 (1417467542776053761), 326 (1417467542777102338), 332 (1417467542780248064)]} 0 61
[junit4:junit4]   2> 6719 T226 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[322 (1417467542764519425), 324 (1417467542764519429), 325 (1417467542766616578), 331 (1417467542767665152), 335 (1417467542768713728), 336 (1417467542769762304), 337 (1417467542770810881), 340 (1417467542771859456), 343 (1417467542772908032), 344 (1417467542780248064)]} 0 17
[junit4:junit4]   2> 6717 T281 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[157 (1417467542695313408), 158 (1417467542696361985), 159 (1417467542701604864), 160 (1417467542761373696), 161 (1417467542768713728), 163 (1417467542770810881), 164 (1417467542772908032), 168 (1417467542775005186), 169 (1417467542777102338), 173 (1417467542778150914)]} 0 80
[junit4:junit4]   2> 6717 T297 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[334 (1417467542764519425), 335 (1417467542765568001), 337 (1417467542766616578), 338 (1417467542768713729), 339 (1417467542770810880), 340 (1417467542771859457), 341 (1417467542773956610), 342 (1417467542775005184), 344 (1417467542777102337), 346 (1417467542778150914)]} 0 15
[junit4:junit4]   2> 6742 T334 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[374 (1417467542785490944), 375 (1417467542789685248), 376 (1417467542790733825), 378 (1417467542791782401), 380 (1417467542792830976), 381 (1417467542795976706), 382 (1417467542799122432), 386 (1417467542800171008), 388 (1417467542801219585), 389 (1417467542804365312)]} 0 19
[junit4:junit4]   2> 6714 T296 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[179 (1417467542750887937), 181 (1417467542751936515), 182 (1417467542752985089), 183 (1417467542755082240), 184 (1417467542756130818), 186 (1417467542758227969), 187 (1417467542760325120), 188 (1417467542771859456), 189 (1417467542772908032), 190 (1417467542773956608)]} 0 72
[junit4:junit4]   2> 6743 T345 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[286 (1417467542778150912), 293 (1417467542779199488), 294 (1417467542791782402), 296 (1417467542795976705), 297 (1417467542798073856), 298 (1417467542799122433), 300 (1417467542800171010), 301 (1417467542801219586), 304 (1417467542802268162), 305 (1417467542805413889)]} 0 27
[junit4:junit4]   2> 6712 T279 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[252 (1417467542751936514), 254 (1417467542754033666), 255 (1417467542755082242), 256 (1417467542756130819), 258 (1417467542757179393), 261 (1417467542758227968), 265 (1417467542759276545), 266 (1417467542760325122), 267 (1417467542769762306), 268 (1417467542770810881)]} 0 21
[junit4:junit4]   2> 6708 T227 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[144 (1417467542679584768), 149 (1417467542682730496), 150 (1417467542682730497), 151 (1417467542683779072), 152 (1417467542684827648), 154 (1417467542691119104), 155 (1417467542700556290), 156 (1417467542701604865), 158 (1417467542761373696), 160 (1417467542768713729)]} 0 87
[junit4:junit4]   2> 6707 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[204 (1417467542700556288), 206 (1417467542702653440), 211 (1417467542705799168), 215 (1417467542707896320), 218 (1417467542708944896), 219 (1417467542709993472), 223 (1417467542762422272), 225 (1417467542763470848), 226 (1417467542765568000), 228 (1417467542766616577)]} 0 65
[junit4:junit4]   2> 6706 T288 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[122 (1417467542572630016), 124 (1417467542575775745), 126 (1417467542676439041), 127 (1417467542678536192), 129 (1417467542680633344), 130 (1417467542686924802), 133 (1417467542687973377), 134 (1417467542695313410), 135 (1417467542761373696), 138 (1417467542764519424)]} 0 186
[junit4:junit4]   2> 6705 T283 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[224 (1417467542696361985), 228 (1417467542699507712), 229 (1417467542700556291), 232 (1417467542701604866), 234 (1417467542702653442), 236 (1417467542704750592), 237 (1417467542704750594), 238 (1417467542762422272), 239 (1417467542763470849), 240 (1417467542765568001)]} 0 67
[junit4:junit4]   2> 6704 T310 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[179 (1417467542702653441), 181 (1417467542707896320), 182 (1417467542707896324), 183 (1417467542713139200), 187 (1417467542714187776), 192 (1417467542715236352), 193 (1417467542715236353), 194 (1417467542716284928), 199 (1417467542717333505), 202 (1417467542763470849)]} 0 60
[junit4:junit4]   2> 6704 T301 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[86 (1417467542677487616), 92 (1417467542678536192), 94 (1417467542681681920), 99 (1417467542689021953), 103 (1417467542695313410), 106 (1417467542696361986), 107 (1417467542700556289), 108 (1417467542761373697), 111 (1417467542763470850), 112 (1417467542764519426)]} 0 84
[junit4:junit4]   2> 6702 T294 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[134 (1417467542678536192), 135 (1417467542681681920), 138 (1417467542682730496), 141 (1417467542686924801), 143 (1417467542689021952), 145 (1417467542690070530), 152 (1417467542695313408), 153 (1417467542696361985), 155 (1417467542703702017), 158 (1417467542762422272)]} 0 82
[junit4:junit4]   2> 6700 T253 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[155 (1417467542683779072), 158 (1417467542686924801), 160 (1417467542689021953), 162 (1417467542717333504), 164 (1417467542751936512), 166 (1417467542754033665), 168 (1417467542756130816), 170 (1417467542757179392), 174 (1417467542759276544), 178 (1417467542760325121)]} 0 75
[junit4:junit4]   2> 6698 T240 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[308 (1417467542717333504), 316 (1417467542751936512), 319 (1417467542752985088), 332 (1417467542752985089), 335 (1417467542754033664), 336 (1417467542755082240), 338 (1417467542756130816), 339 (1417467542756130817), 340 (1417467542757179392), 341 (1417467542758227968)]} 0 41
[junit4:junit4]   2> 6745 T339 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[390 (1417467542772908034), 391 (1417467542773956609), 392 (1417467542776053760), 393 (1417467542777102336), 395 (1417467542778150915), 396 (1417467542793879552), 397 (1417467542795976707), 398 (1417467542799122435), 399 (1417467542800171009), 400 (1417467542807511040)]} 0 34
[junit4:junit4]   2> 6745 T343 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[269 (1417467542770810883), 270 (1417467542776053762), 271 (1417467542778150913), 273 (1417467542780248065), 274 (1417467542781296641), 278 (1417467542792830977), 279 (1417467542794928128), 281 (1417467542804365313), 283 (1417467542805413890), 285 (1417467542806462464)]} 0 36
[junit4:junit4]   2> 6752 T320 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[430 (1417467542808559616), 432 (1417467542808559617), 433 (1417467542809608192), 436 (1417467542810656768), 442 (1417467542810656769), 443 (1417467542811705344), 444 (1417467542812753920), 446 (1417467542812753921), 448 (1417467542813802496), 449 (1417467542814851072)]} 0 8
[junit4:junit4]   2> 6743 T331 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[346 (1417467542780248069), 349 (1417467542782345216), 350 (1417467542785490946), 351 (1417467542791782402), 354 (1417467542793879552), 357 (1417467542797025280), 361 (1417467542799122432), 366 (1417467542800171009), 367 (1417467542801219585), 372 (1417467542805413888)]} 0 25
[junit4:junit4]   2> 6743 T304 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[224 (1417467542765568000), 228 (1417467542766616576), 230 (1417467542769762304), 231 (1417467542770810882), 235 (1417467542775005185), 238 (1417467542785490945), 244 (1417467542794928129), 249 (1417467542797025280), 250 (1417467542802268161), 251 (1417467542805413888)]} 0 40
[junit4:junit4]   2> 6753 T309 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[350 (1417467542764519424), 351 (1417467542765568000), 352 (1417467542766616577), 354 (1417467542785490946), 355 (1417467542791782400), 357 (1417467542793879553), 358 (1417467542794928128), 359 (1417467542797025282), 361 (1417467542802268164), 363 (1417467542815899648)]} 0 51
[junit4:junit4]   2> 6741 T329 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[350 (1417467542781296640), 354 (1417467542782345216), 355 (1417467542783393792), 357 (1417467542790733824), 359 (1417467542791782400), 360 (1417467542793879553), 361 (1417467542795976704), 362 (1417467542799122434), 364 (1417467542801219584), 365 (1417467542802268160)]} 0 22
[junit4:junit4]   2> 6739 T312 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[241 (1417467542769762305), 242 (1417467542779199489), 243 (1417467542780248065), 244 (1417467542781296643), 246 (1417467542792830976), 247 (1417467542793879554), 248 (1417467542795976704), 250 (1417467542797025280), 251 (1417467542800171008), 252 (1417467542801219587)]} 0 31
[junit4:junit4]   2> 6739 T344 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[279 (1417467542775005185), 283 (1417467542777102339), 285 (1417467542779199490), 287 (1417467542783393793), 290 (1417467542786539520), 291 (1417467542792830976), 292 (1417467542794928129), 294 (1417467542798073856), 295 (1417467542800171008), 296 (1417467542801219584)]} 0 26
[junit4:junit4]   2> 6736 T295 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[247 (1417467542771859456), 249 (1417467542773956610), 250 (1417467542777102336), 251 (1417467542780248066), 252 (1417467542783393792), 254 (1417467542784442370), 257 (1417467542785490945), 258 (1417467542791782403), 259 (1417467542793879553), 260 (1417467542795976705)]} 0 27
[junit4:junit4]   2> 6735 T341 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[263 (1417467542767665153), 264 (1417467542769762304), 266 (1417467542773956612), 267 (1417467542776053761), 268 (1417467542778150915), 269 (1417467542779199489), 271 (1417467542781296640), 275 (1417467542784442369), 276 (1417467542792830977), 278 (1417467542795976704)]} 0 29
[junit4:junit4]   2> 6760 T340 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[254 (1417467542767665154), 255 (1417467542777102336), 256 (1417467542780248067), 257 (1417467542781296641), 258 (1417467542793879552), 259 (1417467542794928129), 262 (1417467542801219584), 263 (1417467542802268161), 264 (1417467542802268165), 266 (1417467542821142529)]} 0 54
[junit4:junit4]   2> 6734 T324 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[328 (1417467542764519425), 331 (1417467542766616578), 332 (1417467542767665152), 334 (1417467542769762305), 335 (1417467542771859458), 336 (1417467542773956609), 337 (1417467542776053762), 339 (1417467542780248068), 340 (1417467542782345219), 345 (1417467542783393794)]} 0 32
[junit4:junit4]   2> 6765 T283 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[440 (1417467542810656769), 442 (1417467542816948225), 444 (1417467542817996802), 448 (1417467542824288256), 449 (1417467542824288258), 450 (1417467542825336833), 452 (1417467542826385409), 454 (1417467542827433985), 456 (1417467542827433987), 457 (1417467542828482562)]} 0 18
[junit4:junit4]   2> 6766 T285 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[469 (1417467542815899648), 470 (1417467542816948224), 472 (1417467542817996800), 473 (1417467542819045376), 476 (1417467542822191106), 478 (1417467542823239683), 479 (1417467542824288258), 480 (1417467542827433984), 481 (1417467542827433987), 483 (1417467542828482561)]} 0 14
[junit4:junit4]   2> 6768 T338 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[400 (1417467542801219585), 402 (1417467542802268163), 403 (1417467542804365312), 404 (1417467542817996801), 405 (1417467542819045376), 410 (1417467542826385408), 411 (1417467542827433986), 414 (1417467542828482563), 416 (1417467542829531138), 418 (1417467542831628288)]} 0 31
[junit4:junit4]   2> 6768 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[408 (1417467542820093954), 410 (1417467542821142529), 412 (1417467542822191105), 415 (1417467542823239681), 419 (1417467542825336834), 421 (1417467542826385409), 423 (1417467542827433986), 424 (1417467542829531136), 427 (1417467542830579712), 428 (1417467542831628289)]} 0 12
[junit4:junit4]   2> 6769 T356 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[344 (1417467542821142528), 346 (1417467542822191106), 347 (1417467542825336835), 349 (1417467542826385410), 357 (1417467542827433984), 360 (1417467542829531137), 361 (1417467542830579715), 362 (1417467542831628289), 363 (1417467542832676865), 365 (1417467542833725440)]} 0 12
[junit4:junit4]   2> 6770 T351 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[442 (1417467542807511041), 445 (1417467542809608193), 446 (1417467542810656771), 448 (1417467542811705346), 449 (1417467542813802496), 450 (1417467542814851072), 452 (1417467542815899648), 453 (1417467542816948224), 454 (1417467542817996801), 456 (1417467542831628288)]} 0 26
[junit4:junit4]   2> 6732 T242 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[293 (1417467542768713730), 294 (1417467542770810880), 296 (1417467542771859457), 297 (1417467542775005184), 298 (1417467542776053760), 299 (1417467542778150912), 300 (1417467542779199488), 304 (1417467542780248064), 305 (1417467542784442371), 307 (1417467542792830978)]} 0 25
[junit4:junit4]   2> 6773 T310 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[412 (1417467542811705344), 415 (1417467542812753920), 416 (1417467542813802496), 417 (1417467542815899648), 418 (1417467542816948224), 419 (1417467542817996800), 421 (1417467542824288257), 425 (1417467542825336836), 426 (1417467542829531139), 428 (1417467542836871168)]} 0 25
[junit4:junit4]   2> 6771 T334 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[457 (1417467542810656768), 458 (1417467542811705344), 459 (1417467542812753921), 460 (1417467542813802498), 461 (1417467542814851074), 462 (1417467542815899649), 463 (1417467542817996800), 465 (1417467542821142528), 469 (1417467542822191104), 471 (1417467542833725441)]} 0 25
[junit4:junit4]   2> 6770 T286 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[484 (1417467542815899649), 485 (1417467542817996801), 488 (1417467542820093952), 489 (1417467542823239682), 490 (1417467542825336833), 491 (1417467542826385410), 493 (1417467542828482560), 494 (1417467542829531138), 495 (1417467542830579713), 500 (1417467542833725440)]} 0 18
[junit4:junit4]   2> 6776 T241 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[487 (1417467542828482560), 488 (1417467542830579714), 491 (1417467542831628288), 492 (1417467542832676867), 496 (1417467542835822592), 498 (1417467542836871169), 501 (1417467542837919744), 502 (1417467542838968321), 504 (1417467542838968323), 505 (1417467542840016898)]} 0 13
[junit4:junit4]   2> 6768 T271 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[375 (1417467542820093953), 380 (1417467542821142530), 382 (1417467542822191104), 387 (1417467542823239680), 388 (1417467542824288256), 394 (1417467542825336832), 398 (1417467542826385408), 400 (1417467542827433985), 401 (1417467542829531137), 406 (1417467542829531139)]} 0 12
[junit4:junit4]   2> 6777 T240 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[470 (1417467542821142529), 472 (1417467542822191107), 473 (1417467542823239681), 475 (1417467542826385409), 476 (1417467542829531136), 479 (1417467542829531140), 481 (1417467542837919745), 482 (1417467542838968322), 483 (1417467542840016897), 485 (1417467542841065473)]} 0 20
[junit4:junit4]   2> 6758 T313 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[267 (1417467542767665153), 268 (1417467542775005184), 270 (1417467542777102337), 271 (1417467542781296640), 274 (1417467542782345217), 276 (1417467542784442368), 279 (1417467542785490945), 280 (1417467542799122432), 284 (1417467542803316736), 286 (1417467542821142530)]} 0 53
[junit4:junit4]   2> 6779 T300 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[451 (1417467542821142528), 452 (1417467542824288257), 455 (1417467542831628288), 456 (1417467542832676864), 458 (1417467542835822592), 460 (1417467542836871168), 461 (1417467542837919744), 466 (1417467542838968321), 467 (1417467542840016897), 468 (1417467542843162624)]} 0 23
[junit4:junit4]   2> 6780 T227 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[489 (1417467542823239681), 490 (1417467542825336832), 492 (1417467542829531140), 493 (1417467542831628290), 500 (1417467542836871168), 502 (1417467542837919744), 506 (1417467542838968320), 508 (1417467542840016896), 509 (1417467542842114049), 511 (1417467542844211200)]} 0 21
[junit4:junit4]   2> 6756 T314 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[305 (1417467542764519428), 307 (1417467542782345216), 311 (1417467542783393793), 312 (1417467542787588097), 313 (1417467542788636672), 314 (1417467542789685249), 316 (1417467542797025283), 317 (1417467542802268162), 319 (1417467542807511040), 321 (1417467542817996800)]} 0 53
[junit4:junit4]   2> 6755 T335 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[364 (1417467542783393794), 365 (1417467542784442370), 366 (1417467542786539520), 368 (1417467542788636673), 370 (1417467542789685248), 371 (1417467542790733824), 374 (1417467542800171009), 376 (1417467542801219586), 377 (1417467542807511041), 379 (1417467542816948224)]} 0 35
[junit4:junit4]   2> 6754 T342 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[278 (1417467542768713728), 279 (1417467542784442369), 280 (1417467542788636672), 282 (1417467542791782400), 284 (1417467542792830977), 287 (1417467542797025280), 289 (1417467542799122432), 290 (1417467542800171008), 291 (1417467542802268160), 292 (1417467542816948225)]} 0 47
[junit4:junit4]   2> 6754 T282 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[287 (1417467542766616576), 289 (1417467542767665155), 290 (1417467542776053760), 294 (1417467542777102339), 295 (1417467542783393795), 297 (1417467542785490944), 300 (1417467542797025281), 301 (1417467542800171010), 303 (1417467542802268160), 304 (1417467542807511042)]} 0 50
[junit4:junit4]   2> 6752 T256 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[420 (1417467542807511042), 421 (1417467542808559616), 425 (1417467542809608192), 427 (1417467542810656769), 429 (1417467542810656770), 433 (1417467542811705345), 434 (1417467542812753920), 435 (1417467542812753922), 439 (1417467542813802497), 441 (1417467542814851073)]} 0 8
[junit4:junit4]   2> 6793 T296 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[475 (1417467542836871168), 478 (1417467542836871170), 482 (1417467542837919745), 483 (1417467542838968320), 487 (1417467542840016898), 488 (1417467542841065474), 489 (1417467542842114050), 493 (1417467542843162626), 495 (1417467542845259778), 496 (1417467542857842688)]} 0 22
[junit4:junit4]   2> 6780 T320 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[501 (1417467542836871169), 502 (1417467542837919745), 503 (1417467542838968320), 504 (1417467542840016896), 505 (1417467542841065472), 506 (1417467542842114048), 507 (1417467542842114049), 509 (1417467542843162625), 512 (1417467542844211200), 513 (1417467542845259776)]} 0 8
[junit4:junit4]   2> 6780 T297 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[499 (1417467542836871169), 503 (1417467542836871171), 504 (1417467542837919744), 505 (1417467542840016897), 506 (1417467542841065472), 508 (1417467542841065475), 511 (1417467542842114048), 513 (1417467542843162624), 516 (1417467542844211200), 517 (1417467542845259776)]} 0 9
[junit4:junit4]   2> 6780 T255 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[401 (1417467542820093952), 402 (1417467542831628289), 403 (1417467542833725440), 404 (1417467542840016896), 409 (1417467542841065473), 410 (1417467542841065476), 411 (1417467542842114049), 413 (1417467542843162625), 414 (1417467542844211201), 419 (1417467542845259777)]} 0 24
[junit4:junit4]   2> 6779 T301 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[473 (1417467542821142528), 474 (1417467542822191104), 475 (1417467542823239680), 476 (1417467542824288257), 479 (1417467542825336834), 480 (1417467542829531137), 481 (1417467542833725440), 482 (1417467542834774017), 484 (1417467542843162624), 486 (1417467542844211201)]} 0 22
[junit4:junit4]   2> 6777 T228 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[419 (1417467542810656768), 421 (1417467542821142531), 422 (1417467542825336835), 423 (1417467542827433984), 425 (1417467542828482561), 426 (1417467542830579712), 431 (1417467542831628289), 433 (1417467542832676864), 436 (1417467542834774016), 439 (1417467542841065473)]} 0 31
[junit4:junit4]   2> 6777 T226 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[458 (1417467542828482560), 459 (1417467542829531136), 462 (1417467542829531139), 464 (1417467542830579713), 465 (1417467542831628291), 466 (1417467542832676865), 467 (1417467542833725441), 468 (1417467542833725442), 470 (1417467542838968321), 471 (1417467542841065472)]} 0 14
[junit4:junit4]   2> 6812 T311 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[564 (1417467542866231298), 567 (1417467542867279873), 568 (1417467542868328450), 569 (1417467542869377026), 571 (1417467542870425601), 572 (1417467542871474177), 573 (1417467542872522752), 574 (1417467542873571328), 575 (1417467542876717056), 576 (1417467542877765634)]} 0 12
[junit4:junit4]   2> 6777 T294 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[367 (1417467542820093952), 368 (1417467542821142530), 372 (1417467542823239680), 374 (1417467542825336833), 376 (1417467542827433985), 377 (1417467542830579712), 379 (1417467542836871170), 380 (1417467542838968320), 381 (1417467542840016896), 385 (1417467542841065472)]} 0 21
[junit4:junit4]   2> 6935 T329 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[631 (1417467543001497600), 633 (1417467543002546176), 635 (1417467543003594752), 636 (1417467543003594753), 638 (1417467543003594754), 639 (1417467543004643328), 640 (1417467543004643329), 641 (1417467543005691904), 642 (1417467543005691905), 643 (1417467543006740480)]} 0 6
[junit4:junit4]   2> 6864 T314 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[748 (1417467542897688577), 750 (1417467542908174339), 751 (1417467542913417217), 752 (1417467542921805824), 753 (1417467542926000130), 754 (1417467542928097280), 756 (1417467542929145856), 759 (1417467542930194433), 761 (1417467542931243008), 765 (1417467542932291584)]} 0 34
[junit4:junit4]   2> 6862 T227 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[656 (1417467542896640000), 657 (1417467542909222913), 658 (1417467542914465793), 659 (1417467542915514370), 661 (1417467542915514371), 662 (1417467542920757248), 667 (1417467542926000129), 668 (1417467542927048705), 670 (1417467542928097281), 671 (1417467542930194432)]} 0 34
[junit4:junit4]   2> 6860 T285 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[625 (1417467542896640002), 629 (1417467542908174336), 630 (1417467542909222913), 632 (1417467542912368644), 633 (1417467542914465793), 634 (1417467542915514371), 635 (1417467542918660100), 636 (1417467542922854402), 637 (1417467542927048704), 638 (1417467542928097280)]} 0 31
[junit4:junit4]   2> 6860 T253 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[540 (1417467542883008513), 543 (1417467542909222913), 547 (1417467542910271491), 550 (1417467542911320065), 551 (1417467542918660097), 553 (1417467542922854402), 554 (1417467542923902977), 555 (1417467542924951553), 556 (1417467542927048704), 558 (1417467542928097280)]} 0 44
[junit4:junit4]   2> 6859 T343 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[645 (1417467542896640000), 646 (1417467542907125760), 647 (1417467542908174337), 648 (1417467542910271490), 649 (1417467542914465793), 656 (1417467542915514369), 657 (1417467542922854400), 660 (1417467542923902978), 665 (1417467542926000128), 666 (1417467542927048705)]} 0 30
[junit4:junit4]   2> 6859 T313 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[697 (1417467542883008514), 698 (1417467542907125762), 699 (1417467542909222914), 700 (1417467542912368640), 701 (1417467542914465794), 702 (1417467542919708674), 705 (1417467542920757250), 706 (1417467542921805826), 707 (1417467542926000128), 708 (1417467542927048704)]} 0 43
[junit4:junit4]   2> 6859 T214 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[708 (1417467542898737152), 711 (1417467542910271488), 714 (1417467542912368641), 721 (1417467542913417216), 722 (1417467542915514368), 725 (1417467542917611523), 727 (1417467542920757248), 728 (1417467542922854400), 729 (1417467542923902977), 732 (1417467542924951552)]} 0 28
[junit4:junit4]   2> 6858 T256 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[603 (1417467542898737152), 605 (1417467542906077184), 606 (1417467542908174336), 612 (1417467542909222912), 613 (1417467542909222914), 616 (1417467542910271492), 617 (1417467542911320066), 627 (1417467542922854404), 628 (1417467542923902980), 630 (1417467542926000130)]} 0 27
[junit4:junit4]   2> 6951 T351 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[768 (1417467543013031937), 769 (1417467543014080515), 775 (1417467543015129089), 776 (1417467543016177668), 778 (1417467543017226242), 780 (1417467543018274818), 781 (1417467543020371968), 783 (1417467543021420544), 784 (1417467543022469120), 785 (1417467543023517696)]} 0 10
[junit4:junit4]   2> 6858 T345 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[583 (1417467542875668481), 584 (1417467542878814208), 585 (1417467542879862785), 586 (1417467542881959936), 587 (1417467542906077185), 588 (1417467542921805824), 589 (1417467542922854401), 598 (1417467542923902976), 599 (1417467542923902979), 602 (1417467542926000129)]} 0 49
[junit4:junit4]   2> 6858 T344 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[686 (1417467542898737154), 689 (1417467542907125761), 691 (1417467542908174340), 694 (1417467542910271489), 697 (1417467542911320065), 698 (1417467542912368643), 699 (1417467542918660098), 704 (1417467542921805826), 706 (1417467542923902976), 707 (1417467542924951553)]} 0 26
[junit4:junit4]   2> 6951 T228 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[610 (1417467543013031937), 611 (1417467543015129088), 612 (1417467543016177664), 617 (1417467543016177666), 619 (1417467543018274817), 621 (1417467543020371969), 623 (1417467543021420544), 624 (1417467543022469120), 628 (1417467543023517696), 629 (1417467543023517698)]} 0 11
[junit4:junit4]   2> 6857 T339 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[518 (1417467542880911360), 521 (1417467542883008512), 525 (1417467542910271489), 528 (1417467542911320064), 530 (1417467542918660096), 531 (1417467542919708673), 532 (1417467542920757248), 534 (1417467542921805825), 537 (1417467542922854403), 538 (1417467542924951552)]} 0 42
[junit4:junit4]   2> 6855 T300 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[661 (1417467542898737153), 665 (1417467542909222914), 666 (1417467542911320066), 667 (1417467542913417218), 669 (1417467542914465795), 672 (1417467542916562944), 675 (1417467542917611522), 679 (1417467542918660097), 680 (1417467542921805825), 681 (1417467542922854401)]} 0 24
[junit4:junit4]   2> 6952 T369 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[733 (1417467543014080512), 738 (1417467543015129088), 739 (1417467543015129089), 742 (1417467543016177666), 743 (1417467543016177667), 744 (1417467543017226242), 745 (1417467543018274817), 748 (1417467543019323393), 753 (1417467543023517697), 754 (1417467543024566272)]} 0 11
[junit4:junit4]   2> 6953 T371 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[792 (1417467543016177665), 794 (1417467543017226241), 798 (1417467543018274816), 800 (1417467543019323392), 801 (1417467543020371968), 804 (1417467543020371969), 805 (1417467543022469120), 807 (1417467543023517696), 808 (1417467543024566273), 810 (1417467543025614849)]} 0 11
[junit4:junit4]   2> 6954 T370 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[771 (1417467543016177664), 777 (1417467543017226240), 778 (1417467543017226243), 779 (1417467543018274818), 783 (1417467543019323394), 784 (1417467543021420544), 786 (1417467543022469121), 788 (1417467543023517698), 790 (1417467543025614848), 791 (1417467543026663424)]} 0 12
[junit4:junit4]   2> 6955 T294 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[650 (1417467543015129088), 651 (1417467543016177664), 653 (1417467543017226240), 657 (1417467543018274816), 658 (1417467543019323392), 660 (1417467543020371968), 662 (1417467543022469120), 666 (1417467543023517698), 668 (1417467543025614848), 670 (1417467543026663426)]} 0 14
[junit4:junit4]   2> 6854 T282 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[630 (1417467542898737152), 633 (1417467542908174337), 634 (1417467542910271490), 635 (1417467542913417218), 637 (1417467542914465795), 640 (1417467542916562946), 651 (1417467542918660096), 653 (1417467542919708672), 654 (1417467542920757249), 655 (1417467542921805825)]} 0 24
[junit4:junit4]   2> 6854 T286 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[639 (1417467542896640001), 642 (1417467542908174337), 647 (1417467542908174341), 648 (1417467542912368642), 650 (1417467542913417217), 651 (1417467542917611520), 652 (1417467542917611524), 654 (1417467542918660099), 656 (1417467542919708674), 659 (1417467542921805824)]} 0 25
[junit4:junit4]   2> 6853 T271 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[584 (1417467542896640000), 587 (1417467542908174338), 590 (1417467542910271490), 592 (1417467542912368645), 596 (1417467542914465792), 598 (1417467542915514370), 600 (1417467542916562946), 601 (1417467542917611525), 602 (1417467542919708673), 604 (1417467542920757249)]} 0 24
[junit4:junit4]   2> 6853 T297 C18 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[712 (1417467542910271488), 713 (1417467542910271493), 714 (1417467542914465792), 715 (1417467542915514368), 719 (1417467542915514370), 720 (1417467542916562944), 722 (1417467542917611520), 726 (1417467542919708672), 727 (1417467542919708674), 736 (1417467542920757249)]} 0 12
[junit4:junit4]   2> 6853 T342 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[694 (1417467542884057091), 696 (1417467542906077185), 702 (1417467542907125760), 705 (1417467542908174337), 707 (1417467542909222912), 709 (1417467542915514368), 710 (1417467542916562944), 716 (1417467542917611520), 720 (1417467542918660096), 721 (1417467542919708672)]} 0 37
[junit4:junit4]   2> 6852 T335 C16 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[710 (1417467542897688576), 712 (1417467542907125761), 713 (1417467542913417216), 714 (1417467542914465792), 716 (1417467542915514369), 718 (1417467542916562944), 719 (1417467542916562947), 720 (1417467542917611520), 721 (1417467542918660097), 722 (1417467542919708673)]} 0 22
[junit4:junit4]   2> 6851 T320 C15 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[755 (1417467542910271491), 756 (1417467542912368640), 757 (1417467542912368646), 759 (1417467542913417219), 763 (1417467542914465794), 764 (1417467542915514369), 766 (1417467542916562945), 767 (1417467542917611521), 768 (1417467542918660096), 770 (1417467542919708672)]} 0 9
[junit4:junit4]   2> 6850 T241 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[597 (1417467542883008512), 599 (1417467542884057090), 600 (1417467542884057093), 601 (1417467542910271488), 603 (1417467542911320064), 606 (1417467542912

[...truncated too long message...]

eadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=183, name=qtp664118822-183 Acceptor2 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=181, name=qtp664118822-181 Acceptor0 SelectChannelConnector@0.0.0.0:52944, state=RUNNABLE, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([429B0813EFF4D0E]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=91, name=qtp106686383-91 Acceptor1 SelectChannelConnector@0.0.0.0:52938, state=RUNNABLE, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=93, name=qtp106686383-93 Acceptor3 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=90, name=qtp106686383-90 Acceptor0 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=92, name=qtp106686383-92 Acceptor2 SelectChannelConnector@0.0.0.0:52938, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    5) Thread[id=182, name=qtp664118822-182 Acceptor1 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    6) Thread[id=184, name=qtp664118822-184 Acceptor3 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=183, name=qtp664118822-183 Acceptor2 SelectChannelConnector@0.0.0.0:52944, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=181, name=qtp664118822-181 Acceptor0 SelectChannelConnector@0.0.0.0:52944, state=RUNNABLE, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([429B0813EFF4D0E]:0)
[junit4:junit4] Completed on J1 in 87.82s, 1 test, 2 errors <<< FAILURES!

[...truncated 170 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 771 ignored (2 assumptions)

Total time: 18 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 828 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/828/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedQueryElevationComponentTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedQueryElevationComponentTest:     1) Thread[id=43, name=qtp551596502-43 Acceptor0 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=85, name=qtp659341119-85 Acceptor0 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=58, name=qtp758655368-58 Acceptor1 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=74, name=qtp1791077327-74 Acceptor3 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=73, name=qtp1791077327-73 Acceptor2 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=71, name=qtp1791077327-71 Acceptor0 SelectChannelConnector@0.0.0.0:14245, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=44, name=qtp551596502-44 Acceptor1 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=57, name=qtp758655368-57 Acceptor0 SelectChannelConnector@0.0.0.0:27312, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=87, name=qtp659341119-87 Acceptor2 SelectChannelConnector@0.0.0.0:43605, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=88, name=qtp659341119-88 Acceptor3 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=60, name=qtp758655368-60 Acceptor3 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=86, name=qtp659341119-86 Acceptor1 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=45, name=qtp551596502-45 Acceptor2 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=59, name=qtp758655368-59 Acceptor2 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=72, name=qtp1791077327-72 Acceptor1 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=46, name=qtp551596502-46 Acceptor3 SelectChannelConnector@0.0.0.0:15984, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedQueryElevationComponentTest: 
   1) Thread[id=43, name=qtp551596502-43 Acceptor0 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=85, name=qtp659341119-85 Acceptor0 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=58, name=qtp758655368-58 Acceptor1 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=74, name=qtp1791077327-74 Acceptor3 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=73, name=qtp1791077327-73 Acceptor2 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=71, name=qtp1791077327-71 Acceptor0 SelectChannelConnector@0.0.0.0:14245, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=44, name=qtp551596502-44 Acceptor1 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=57, name=qtp758655368-57 Acceptor0 SelectChannelConnector@0.0.0.0:27312, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=87, name=qtp659341119-87 Acceptor2 SelectChannelConnector@0.0.0.0:43605, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=88, name=qtp659341119-88 Acceptor3 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=60, name=qtp758655368-60 Acceptor3 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=86, name=qtp659341119-86 Acceptor1 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=45, name=qtp551596502-45 Acceptor2 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=59, name=qtp758655368-59 Acceptor2 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=72, name=qtp1791077327-72 Acceptor1 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=46, name=qtp551596502-46 Acceptor3 SelectChannelConnector@0.0.0.0:15984, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([9C810015AEC1D846]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedQueryElevationComponentTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=43, name=qtp551596502-43 Acceptor0 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=85, name=qtp659341119-85 Acceptor0 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=58, name=qtp758655368-58 Acceptor1 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=74, name=qtp1791077327-74 Acceptor3 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=71, name=qtp1791077327-71 Acceptor0 SelectChannelConnector@0.0.0.0:14245, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=73, name=qtp1791077327-73 Acceptor2 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=44, name=qtp551596502-44 Acceptor1 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=57, name=qtp758655368-57 Acceptor0 SelectChannelConnector@0.0.0.0:27312, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=87, name=qtp659341119-87 Acceptor2 SelectChannelConnector@0.0.0.0:43605, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=88, name=qtp659341119-88 Acceptor3 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=60, name=qtp758655368-60 Acceptor3 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=86, name=qtp659341119-86 Acceptor1 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=45, name=qtp551596502-45 Acceptor2 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=59, name=qtp758655368-59 Acceptor2 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=72, name=qtp1791077327-72 Acceptor1 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=46, name=qtp551596502-46 Acceptor3 SelectChannelConnector@0.0.0.0:15984, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=43, name=qtp551596502-43 Acceptor0 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=85, name=qtp659341119-85 Acceptor0 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=58, name=qtp758655368-58 Acceptor1 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=74, name=qtp1791077327-74 Acceptor3 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=71, name=qtp1791077327-71 Acceptor0 SelectChannelConnector@0.0.0.0:14245, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=73, name=qtp1791077327-73 Acceptor2 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=44, name=qtp551596502-44 Acceptor1 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=57, name=qtp758655368-57 Acceptor0 SelectChannelConnector@0.0.0.0:27312, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=87, name=qtp659341119-87 Acceptor2 SelectChannelConnector@0.0.0.0:43605, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=88, name=qtp659341119-88 Acceptor3 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=60, name=qtp758655368-60 Acceptor3 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=86, name=qtp659341119-86 Acceptor1 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=45, name=qtp551596502-45 Acceptor2 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=59, name=qtp758655368-59 Acceptor2 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=72, name=qtp1791077327-72 Acceptor1 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=46, name=qtp551596502-46 Acceptor3 SelectChannelConnector@0.0.0.0:15984, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([9C810015AEC1D846]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedTermsComponentTest

Error Message:
36 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedTermsComponentTest:     1) Thread[id=14, name=qtp1159656515-14 Acceptor3 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=30, name=qtp1674485910-30 Acceptor0 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=134, name=qtp1398945824-134 Acceptor2 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=50, name=qtp24862902-50 Acceptor3 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=133, name=qtp1398945824-133 Acceptor1 SelectChannelConnector@0.0.0.0:50290, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=96, name=qtp1877581236-96 Acceptor3 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=122, name=qtp203292186-122 Acceptor3 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=64, name=qtp2022619994-64 Acceptor3 SelectChannelConnector@0.0.0.0:62126, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=33, name=qtp1674485910-33 Acceptor3 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=121, name=qtp203292186-121 Acceptor2 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=135, name=qtp1398945824-135 Acceptor3 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=120, name=qtp203292186-120 Acceptor1 SelectChannelConnector@0.0.0.0:41313, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=73, name=qtp597360932-73 Acceptor0 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=20, name=qtp1159656515-20 Acceptor0 SelectChannelConnector@0.0.0.0:11228, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=109, name=qtp784663905-109 Acceptor2 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=74, name=qtp597360932-74 Acceptor1 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=31, name=qtp1674485910-31 Acceptor1 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=132, name=qtp1398945824-132 Acceptor0 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=107, name=qtp784663905-107 Acceptor0 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=75, name=qtp597360932-75 Acceptor2 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=119, name=qtp203292186-119 Acceptor0 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=93, name=qtp1877581236-93 Acceptor0 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=62, name=qtp2022619994-62 Acceptor1 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=108, name=qtp784663905-108 Acceptor1 SelectChannelConnector@0.0.0.0:19145, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=17, name=qtp1159656515-17 Acceptor1 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=94, name=qtp1877581236-94 Acceptor1 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=32, name=qtp1674485910-32 Acceptor2 SelectChannelConnector@0.0.0.0:63331, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=95, name=qtp1877581236-95 Acceptor2 SelectChannelConnector@0.0.0.0:44860, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=16, name=qtp1159656515-16 Acceptor2 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=61, name=qtp2022619994-61 Acceptor0 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=47, name=qtp24862902-47 Acceptor0 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=49, name=qtp24862902-49 Acceptor2 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=63, name=qtp2022619994-63 Acceptor2 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=76, name=qtp597360932-76 Acceptor3 SelectChannelConnector@0.0.0.0:39798, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=110, name=qtp784663905-110 Acceptor3 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=48, name=qtp24862902-48 Acceptor1 SelectChannelConnector@0.0.0.0:19293, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 36 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedTermsComponentTest: 
   1) Thread[id=14, name=qtp1159656515-14 Acceptor3 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=30, name=qtp1674485910-30 Acceptor0 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=134, name=qtp1398945824-134 Acceptor2 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=50, name=qtp24862902-50 Acceptor3 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=133, name=qtp1398945824-133 Acceptor1 SelectChannelConnector@0.0.0.0:50290, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=96, name=qtp1877581236-96 Acceptor3 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=122, name=qtp203292186-122 Acceptor3 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=64, name=qtp2022619994-64 Acceptor3 SelectChannelConnector@0.0.0.0:62126, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=33, name=qtp1674485910-33 Acceptor3 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=121, name=qtp203292186-121 Acceptor2 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=135, name=qtp1398945824-135 Acceptor3 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=120, name=qtp203292186-120 Acceptor1 SelectChannelConnector@0.0.0.0:41313, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=73, name=qtp597360932-73 Acceptor0 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=20, name=qtp1159656515-20 Acceptor0 SelectChannelConnector@0.0.0.0:11228, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=109, name=qtp784663905-109 Acceptor2 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=74, name=qtp597360932-74 Acceptor1 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=31, name=qtp1674485910-31 Acceptor1 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=132, name=qtp1398945824-132 Acceptor0 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=107, name=qtp784663905-107 Acceptor0 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=75, name=qtp597360932-75 Acceptor2 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=119, name=qtp203292186-119 Acceptor0 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=93, name=qtp1877581236-93 Acceptor0 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=62, name=qtp2022619994-62 Acceptor1 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=108, name=qtp784663905-108 Acceptor1 SelectChannelConnector@0.0.0.0:19145, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=17, name=qtp1159656515-17 Acceptor1 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=94, name=qtp1877581236-94 Acceptor1 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=32, name=qtp1674485910-32 Acceptor2 SelectChannelConnector@0.0.0.0:63331, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=95, name=qtp1877581236-95 Acceptor2 SelectChannelConnector@0.0.0.0:44860, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=16, name=qtp1159656515-16 Acceptor2 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=61, name=qtp2022619994-61 Acceptor0 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=47, name=qtp24862902-47 Acceptor0 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=49, name=qtp24862902-49 Acceptor2 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=63, name=qtp2022619994-63 Acceptor2 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=76, name=qtp597360932-76 Acceptor3 SelectChannelConnector@0.0.0.0:39798, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=110, name=qtp784663905-110 Acceptor3 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=48, name=qtp24862902-48 Acceptor1 SelectChannelConnector@0.0.0.0:19293, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([9C810015AEC1D846]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedTermsComponentTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=14, name=qtp1159656515-14 Acceptor3 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=30, name=qtp1674485910-30 Acceptor0 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=134, name=qtp1398945824-134 Acceptor2 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=50, name=qtp24862902-50 Acceptor3 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=133, name=qtp1398945824-133 Acceptor1 SelectChannelConnector@0.0.0.0:50290, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=96, name=qtp1877581236-96 Acceptor3 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=122, name=qtp203292186-122 Acceptor3 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=64, name=qtp2022619994-64 Acceptor3 SelectChannelConnector@0.0.0.0:62126, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=33, name=qtp1674485910-33 Acceptor3 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=121, name=qtp203292186-121 Acceptor2 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=135, name=qtp1398945824-135 Acceptor3 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=120, name=qtp203292186-120 Acceptor1 SelectChannelConnector@0.0.0.0:41313, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=73, name=qtp597360932-73 Acceptor0 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=20, name=qtp1159656515-20 Acceptor0 SelectChannelConnector@0.0.0.0:11228, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=74, name=qtp597360932-74 Acceptor1 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=109, name=qtp784663905-109 Acceptor2 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=31, name=qtp1674485910-31 Acceptor1 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=132, name=qtp1398945824-132 Acceptor0 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=75, name=qtp597360932-75 Acceptor2 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=107, name=qtp784663905-107 Acceptor0 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=93, name=qtp1877581236-93 Acceptor0 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=119, name=qtp203292186-119 Acceptor0 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=62, name=qtp2022619994-62 Acceptor1 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=108, name=qtp784663905-108 Acceptor1 SelectChannelConnector@0.0.0.0:19145, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=17, name=qtp1159656515-17 Acceptor1 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=94, name=qtp1877581236-94 Acceptor1 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=32, name=qtp1674485910-32 Acceptor2 SelectChannelConnector@0.0.0.0:63331, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=95, name=qtp1877581236-95 Acceptor2 SelectChannelConnector@0.0.0.0:44860, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=16, name=qtp1159656515-16 Acceptor2 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=61, name=qtp2022619994-61 Acceptor0 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=47, name=qtp24862902-47 Acceptor0 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=49, name=qtp24862902-49 Acceptor2 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=63, name=qtp2022619994-63 Acceptor2 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=76, name=qtp597360932-76 Acceptor3 SelectChannelConnector@0.0.0.0:39798, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=110, name=qtp784663905-110 Acceptor3 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=48, name=qtp24862902-48 Acceptor1 SelectChannelConnector@0.0.0.0:19293, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=14, name=qtp1159656515-14 Acceptor3 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=30, name=qtp1674485910-30 Acceptor0 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=134, name=qtp1398945824-134 Acceptor2 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=50, name=qtp24862902-50 Acceptor3 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=133, name=qtp1398945824-133 Acceptor1 SelectChannelConnector@0.0.0.0:50290, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=96, name=qtp1877581236-96 Acceptor3 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=122, name=qtp203292186-122 Acceptor3 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=64, name=qtp2022619994-64 Acceptor3 SelectChannelConnector@0.0.0.0:62126, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=33, name=qtp1674485910-33 Acceptor3 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=121, name=qtp203292186-121 Acceptor2 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=135, name=qtp1398945824-135 Acceptor3 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=120, name=qtp203292186-120 Acceptor1 SelectChannelConnector@0.0.0.0:41313, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=73, name=qtp597360932-73 Acceptor0 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=20, name=qtp1159656515-20 Acceptor0 SelectChannelConnector@0.0.0.0:11228, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=74, name=qtp597360932-74 Acceptor1 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=109, name=qtp784663905-109 Acceptor2 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=31, name=qtp1674485910-31 Acceptor1 SelectChannelConnector@0.0.0.0:63331, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=132, name=qtp1398945824-132 Acceptor0 SelectChannelConnector@0.0.0.0:50290, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=75, name=qtp597360932-75 Acceptor2 SelectChannelConnector@0.0.0.0:39798, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=107, name=qtp784663905-107 Acceptor0 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=93, name=qtp1877581236-93 Acceptor0 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=119, name=qtp203292186-119 Acceptor0 SelectChannelConnector@0.0.0.0:41313, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=62, name=qtp2022619994-62 Acceptor1 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=108, name=qtp784663905-108 Acceptor1 SelectChannelConnector@0.0.0.0:19145, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=17, name=qtp1159656515-17 Acceptor1 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=94, name=qtp1877581236-94 Acceptor1 SelectChannelConnector@0.0.0.0:44860, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=32, name=qtp1674485910-32 Acceptor2 SelectChannelConnector@0.0.0.0:63331, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=95, name=qtp1877581236-95 Acceptor2 SelectChannelConnector@0.0.0.0:44860, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=16, name=qtp1159656515-16 Acceptor2 SelectChannelConnector@0.0.0.0:11228, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=61, name=qtp2022619994-61 Acceptor0 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=47, name=qtp24862902-47 Acceptor0 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=49, name=qtp24862902-49 Acceptor2 SelectChannelConnector@0.0.0.0:19293, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=63, name=qtp2022619994-63 Acceptor2 SelectChannelConnector@0.0.0.0:62126, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=76, name=qtp597360932-76 Acceptor3 SelectChannelConnector@0.0.0.0:39798, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=110, name=qtp784663905-110 Acceptor3 SelectChannelConnector@0.0.0.0:19145, state=BLOCKED, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=48, name=qtp24862902-48 Acceptor1 SelectChannelConnector@0.0.0.0:19293, state=RUNNABLE, group=TGRP-DistributedTermsComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([9C810015AEC1D846]:0)




Build Log:
[...truncated 7641 lines...]
[junit4:junit4] Suite: org.apache.solr.handler.component.DistributedTermsComponentTest
[junit4:junit4]   2> 212 T10 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 667 T10 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 942 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:11228
[junit4:junit4]   2> 1116 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1116 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1117 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1961 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1967 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1967 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2042 T10 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2048 T10 oasc.CoreContainer.<init> New CoreContainer 1791392392
[junit4:junit4]   2> 2049 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2050 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2144 T10 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2224 T10 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2225 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2254 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2255 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2459 T10 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2667 T10 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2706 T10 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2767 T10 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 5050 T10 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5081 T10 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5085 T10 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5452 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5467 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5471 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5492 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5606 T10 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedTermsComponentTest-1351784111238/control/data/
[junit4:junit4]   2> 5612 T10 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@63ce15f6
[junit4:junit4]   2> 5871 T10 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedTermsComponentTest-1351784111238/control/data/index/
[junit4:junit4]   2> 5872 T10 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedTermsComponentTest-1351784111238/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5975 T10 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedTermsComponentTest-1351784111238/control/data/index forceNew:false
[junit4:junit4]   2> 6429 T10 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@206cc2a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@58d51a54),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 6430 T10 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 6500 T10 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 7713 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 7714 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 7715 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 7723 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 7869 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 7879 T10 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 7884 T10 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 7908 T10 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 7929 T10 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 8006 T10 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 8008 T10 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 8013 T10 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 8014 T10 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 8014 T10 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 8015 T10 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 8045 T10 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 8046 T10 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8047 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8048 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8049 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8050 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8051 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8052 T10 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8073 T10 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 8091 T10 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 8235 T10 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 8269 T10 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 8282 T10 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-REPLICATION2.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-MINIMAL.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-IB.XML, SCHEMA-CHARFILTERS.XML, SOLRCONFIG-QUERYSENDER.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, DA_UTF8.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-SPELLCHECKER.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, SCHEMA-XINCLUDE.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, XSLT, SOLRCONFIG-NATIVELOCK.XML, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-ALTDIRECTORY.XML, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, SOLRCONFIG_PERF.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 8365 T10 oass.SolrIndexSearcher.<init> Opening Searcher@37c2ec05 main
[junit4:junit4]   2> 8366 T10 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 8462 T10 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 8463 T10 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 8465 T10 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 8679 T10 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 9096 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 9097 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 9098 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 9099 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 9100 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 9101 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 9101 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 9102 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 9103 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 9150 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 9594 T24 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37c2ec05 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 9602 T10 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 9602 T10 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 9603 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 9652 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 9674 T10 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 9677 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:63331
[junit4:junit4]   2> 9678 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 9679 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 9679 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 9708 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 9709 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 9709 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 9710 T10 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 9710 T10 oasc.CoreContainer.<init> New CoreContainer 1852317112
[junit4:junit4]   2> 9711 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 9712 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 9735 T10 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 9755 T10 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 9756 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 9758 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 9758 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9834 T10 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 9900 T10 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 9901 T10 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 9938 T10 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 10457 T10 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10470 T10 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10474 T10 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10491 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10496 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10500 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10503 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10506 T10 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedTermsComponentTest-1351784111238/shard0/data/
[junit4:junit4]   2> 10506 T10 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@63ce15f6
[junit4:junit4]   2> 10508 T10 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedTermsComponentTest-1351784111238/shard0/data/index/
[junit4:junit4]   2> 10509 T10 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedTermsComponentTest-1351784111238/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10510 T10 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedTermsComponentTest-1351784111238/shard0/data/index forceNew:false
[junit4:junit4]   2> 10513 T10 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6b64bff9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@59d794d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10514 T10 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10515 T10 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 10518 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10518 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 10519 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10520 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 10520 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10521 T10 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10522 T10 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10523 T10 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10524 T10 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10525 T10 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 10527 T10 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 10527 T10 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10529 T10 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 10529 T10 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 10530 T10 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 10531 T10 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10532 T10 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10534 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10535 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10536 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10537 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10538 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10540 T10 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10541 T10 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 10542 T10 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 10548 T10 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10552 T10 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 10555 T10 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-REPLICATION2.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, SCHEMA-MINIMAL.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, CURRENCY.XML, SOLRCONFIG-MERGEPOLICY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, ELEVATE.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-IB.XML, SCHEMA-CHARFILTERS.XML, SOLRCONFIG-QUERYSENDER.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, DA_UTF8.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, HYPHENATION.DTD, SOLRCONFIG-ENABLEPLUGIN.XML, STEMDICT.TXT, SCHEMA-PHRASESUGGEST.XML, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, STOPTYPES-2.TXT, SCHEMA-REVERSED.XML, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-SPELLCHECKER.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, SCHEMA-XINCLUDE.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA.XML, SCHEMA_CODEC.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, XSLT, SOLRCONFIG-NATIVELOCK.XML, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-ALTDIRECTORY.XML, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG-INDEXCONFIG.XML, SOLRCONFIG_PERF.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 10559 T10 oass.SolrIndexSearcher.<init> Opening Searcher@6db248c main
[junit4:junit4]   2> 10559 T10 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 10560 T10 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10561 T10 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10562 T10 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 10610 T10 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 10655 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 10656 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 10656 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 10657 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 10657 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 10658 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 10658 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 10659 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 10659 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 10660 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 10685 T38 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6db248c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10693 T10 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 10693 T10 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 10694 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 10694 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@14eac7c8
[junit4:junit4]   2> 12858 T21 C0 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 12865 T21 C0 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@206cc2a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@58d51a54),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 12865 T21 C0 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 12871 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 738
[junit4:junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@4ce76438
[junit4:junit4]   2> 13052 T36 C1 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 13055 T36 C1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6b64bff9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@59d794d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 13055 T36 C1 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 13056 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 5
[junit4:junit4]   2> 13358 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 293
[junit4:junit4]   2> 13365 T37 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 3
[junit4:junit4]   2> 13371 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 2
[junit4:junit4]   2> 13378 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 3
[junit4:junit4]   2> 13384 T23 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 2
[junit4:junit4]   2> 13389 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 2
[junit4:junit4]   2> 13394 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 2
[junit4:junit4]   2> 13399 T37 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 2
[junit4:junit4]   2> 13404 T23 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 2
[junit4:junit4]   2> 13408 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 2
[junit4:junit4]   2> 13413 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 1
[junit4:junit4]   2> 13417 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 2
[junit4:junit4]   2> 13421 T23 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 2
[junit4:junit4]   2> 13425 T37 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 1
[junit4:junit4]   2> 13430 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 2
[junit4:junit4]   2> 13435 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 2
[junit4:junit4]   2> 13468 T23 C0 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 14190 T23 C0 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@206cc2a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@58d51a54),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@206cc2a4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@58d51a54),segFN=segments_2,generation=2,filenames=[_0_MockVariableIntBlock_0.doc, _0_Pulsing41_1.tip, _0_MockVariableIntBlock_0.skp, _0_NestedPulsing_0.tip, _0_NestedPulsing_0.tim, _0_Pulsing41_1.tim, _0_MockVariableIntBlock_0.pyl, _0_NestedPulsing_0.pos, _0_MockVariableIntBlock_0.tib, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _0_NestedPulsing_0.doc, _0.fnm, _0_nrm.cfs, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_nrm.cfe, _0_MockVariableIntBlock_0.frq, segments_2, _0_Pulsing41_1.doc, _0.fdx, _0_Pulsing41_1.pos, _0.fdt]
[junit4:junit4]   2> 14191 T23 C0 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 14397 T23 C0 oass.SolrIndexSearcher.<init> Opening Searcher@2f152c0e main
[junit4:junit4]   2> 14398 T23 C0 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 14401 T23 C0 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 14412 T24 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2f152c0e main{StandardDirectoryReader(segments_2:4 _0(4.1):C8)}
[junit4:junit4]   2> 14413 T23 C0 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 949
[junit4:junit4]   2> 14418 T36 C1 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 14428 T36 C1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6b64bff9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@59d794d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6b64bff9 lockFactory=org.apache.lucene.store.NativeFSLockFactory@59d794d),segFN=segments_2,generation=2,filenames=[_0_MockVariableIntBlock_0.doc, _0_Pulsing41_1.tip, _0_MockVariableIntBlock_0.skp, _0_NestedPulsing_0.tip, _0_NestedPulsing_0.tim, _0_Pulsing41_1.tim, _0_MockVariableIntBlock_0.pyl, _0_NestedPulsing_0.pos, _0_MockVariableIntBlock_0.tib, _0_Pulsing41_0.tim, _0_Pulsing41_0.tip, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _0_NestedPulsing_0.doc, _0.fnm, _0_nrm.cfs, _0_Pulsing41_0.pos, _0_Pulsing41_0.doc, _0_nrm.cfe, _0_MockVariableIntBlock_0.frq, segments_2, _0_Pulsing41_1.doc, _0.fdx, _0_Pulsing41_1.pos, _0.fdt]
[junit4:junit4]   2> 14429 T36 C1 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 14435 T36 C1 oass.SolrIndexSearcher.<init> Opening Searcher@483bead5 main
[junit4:junit4]   2> 14435 T36 C1 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 14436 T36 C1 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 14441 T38 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@483bead5 main{StandardDirectoryReader(segments_2:4 _0(4.1):C8)}
[junit4:junit4]   2> 14442 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 24
[junit4:junit4]   2> 14554 T21 C0 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms.fl=b_t&terms=true&distrib=false&qt=/terms&wt=javabin&version=2} status=0 QTime=76 
[junit4:junit4]   2> 14601 T36 C1 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms=true&distrib=false&wt=javabin&terms.sort=index&version=2&shard.url=127.0.0.1:63331/solr|[ff01::114]:33332/solr&NOW=1351784125590&terms.fl=b_t&terms.limit=-1&qt=/terms&isShard=true} status=0 QTime=1 
[junit4:junit4]   2> 14609 T34 C1 REQ [collection1] webapp=/solr path=/terms params={shards=127.0.0.1:63331/solr|[ff01::114]:33332/solr&shards.qt=/terms&terms.fl=b_t&terms=true&qt=/terms&wt=javabin&version=2} status=0 QTime=40 
[junit4:junit4]   2> 14617 T23 C0 REQ [collection1] webapp=/solr path=/terms params={terms.lower=s&shards.qt=/terms&terms.fl=b_t&terms=true&distrib=false&terms.limit=5&qt=/terms&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 14795 T36 C1 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms=true&distrib=false&wt=javabin&terms.sort=index&version=2&NOW=1351784125641&shard.url=127.0.0.1:63331/solr|[ff01::213]:33332/solr|[ff01::114]:33332/solr|[ff01::083]:33332/solr&terms.lower=s&terms.fl=b_t&terms.limit=-1&isShard=true&qt=/terms} status=0 QTime=1 
[junit4:junit4]   2> 14796 T37 C1 REQ [collection1] webapp=/solr path=/terms params={shards=127.0.0.1:63331/solr|[ff01::213]:33332/solr|[ff01::114]:33332/solr|[ff01::083]:33332/solr&terms.lower=s&shards.qt=/terms&terms.fl=b_t&terms=true&terms.limit=5&qt=/terms&wt=javabin&version=2} status=0 QTime=176 
[junit4:junit4]   2> 14801 T39 C0 REQ [collection1] webapp=/solr path=/terms params={terms.lower=sn&shards.qt=/terms&terms.fl=b_t&terms=true&distrib=false&terms.limit=5&qt=/terms&wt=javabin&terms.prefix=sn&version=2} status=0 QTime=1 
[junit4:junit4]   2> 14808 T37 C1 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms=true&distrib=false&wt=javabin&terms.sort=index&terms.prefix=sn&version=2&shard.url=127.0.0.1:63331/solr&NOW=1351784125826&terms.lower=sn&terms.fl=b_t&terms.limit=-1&qt=/terms&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 14810 T36 C1 REQ [collection1] webapp=/solr path=/terms params={shards=127.0.0.1:63331/solr&terms.lower=sn&shards.qt=/terms&terms.fl=b_t&terms=true&terms.limit=5&qt=/terms&wt=javabin&terms.prefix=sn&version=2} status=0 QTime=5 
[junit4:junit4]   2> 14814 T23 C0 REQ [collection1] webapp=/solr path=/terms params={terms.lower=s&shards.qt=/terms&terms.fl=b_t&terms=true&distrib=false&terms.limit=5&qt=/terms&wt=javabin&terms.prefix=s&terms.upper=sn&version=2} status=0 QTime=1 
[junit4:junit4]   2> 14821 T37 C1 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms=true&distrib=false&wt=javabin&terms.sort=index&terms.prefix=s&terms.upper=sn&version=2&shard.url=127.0.0.1:63331/solr&NOW=1351784125838&terms.lower=s&terms.fl=b_t&terms.limit=-1&qt=/terms&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 14822 T34 C1 REQ [collection1] webapp=/solr path=/terms params={shards=127.0.0.1:63331/solr&terms.lower=s&shards.qt=/terms&terms.fl=b_t&terms=true&terms.limit=5&qt=/terms&wt=javabin&terms.prefix=s&terms.upper=sn&version=2} status=0 QTime=5 
[junit4:junit4]   2> 14827 T21 C0 REQ [collection1] webapp=/solr path=/terms params={terms.lower=s&shards.qt=/terms&terms.fl=b_t&terms=true&distrib=false&terms.limit=5&qt=/terms&wt=javabin&terms.sort=index&terms.prefix=s&version=2} status=0 QTime=1 
[junit4:junit4]   2> 14835 T37 C1 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms=true&distrib=false&wt=javabin&terms.sort=index&terms.prefix=s&version=2&shard.url=[ff01::114]:33332/solr|127.0.0.1:63331/solr|[ff01::213]:33332/solr&NOW=1351784125852&terms.lower=s&terms.fl=b_t&terms.limit=-1&qt=/terms&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 14843 T36 C1 REQ [collection1] webapp=/solr path=/terms params={shards=[ff01::114]:33332/solr|127.0.0.1:63331/solr|[ff01::213]:33332/solr&terms.lower=s&shards.qt=/terms&terms.fl=b_t&terms=true&terms.limit=5&qt=/terms&wt=javabin&terms.sort=index&terms.prefix=s&version=2} status=0 QTime=12 
[junit4:junit4]   2> 14847 T39 C0 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms=true&distrib=false&wt=javabin&terms.sort=index&terms.prefix=s&terms.upper=sn&version=2&terms.lower=s&terms.fl=b_t&terms.limit=5&qt=/terms} status=0 QTime=1 
[junit4:junit4]   2> 14854 T37 C1 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms=true&distrib=false&wt=javabin&terms.sort=index&terms.prefix=s&terms.upper=sn&version=2&shard.url=127.0.0.1:63331/solr&NOW=1351784125871&terms.lower=s&terms.fl=b_t&terms.limit=-1&qt=/terms&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 14855 T34 C1 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms=true&wt=javabin&terms.sort=index&terms.prefix=s&terms.upper=sn&version=2&shards=127.0.0.1:63331/solr&terms.lower=s&terms.fl=b_t&terms.limit=5&qt=/terms} status=0 QTime=5 
[junit4:junit4]   2> 14859 T23 C0 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms.fl=b_t&terms=true&distrib=false&qt=/terms&wt=javabin&terms.sort=index&version=2} status=0 QTime=1 
[junit4:junit4]   2> 14866 T37 C1 REQ [collection1] webapp=/solr path=/terms params={shards.qt=/terms&terms=true&distrib=false&wt=javabin&terms.sort=index&version=2&shard.url=127.0.0.1:63331/solr&NOW=1351784125883&terms.fl=b_t&terms.limit=-1&qt=/terms&isShard=true} status=0 QTime=0 
[junit4:junit4]   2> 14867 T36 C1 REQ [collection1] webapp=/solr path=/terms params={shards=127.0.0.1:63331/solr&shards.qt=/terms&terms.fl=b_t&terms=true&qt=/terms&wt=javabin&terms.sort=index&version=2} status=0 QTime=5 
[junit4:junit4]   2> 14914 T10 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1791392392
[junit4:junit4]   2> 14914 T10 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@14eac7c8
[junit4:junit4]   2> 14926 T10 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=8,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 14927 T10 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 14927 T10 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 14928 T10 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 14930 T10 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 15158 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 15260 T10 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be st

[...truncated too long message...]

ead.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=44, name=qtp551596502-44 Acceptor1 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=57, name=qtp758655368-57 Acceptor0 SelectChannelConnector@0.0.0.0:27312, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    9) Thread[id=87, name=qtp659341119-87 Acceptor2 SelectChannelConnector@0.0.0.0:43605, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   10) Thread[id=88, name=qtp659341119-88 Acceptor3 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=60, name=qtp758655368-60 Acceptor3 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=86, name=qtp659341119-86 Acceptor1 SelectChannelConnector@0.0.0.0:43605, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=45, name=qtp551596502-45 Acceptor2 SelectChannelConnector@0.0.0.0:15984, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=59, name=qtp758655368-59 Acceptor2 SelectChannelConnector@0.0.0.0:27312, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=72, name=qtp1791077327-72 Acceptor1 SelectChannelConnector@0.0.0.0:14245, state=BLOCKED, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=46, name=qtp551596502-46 Acceptor3 SelectChannelConnector@0.0.0.0:15984, state=RUNNABLE, group=TGRP-DistributedQueryElevationComponentTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([9C810015AEC1D846]:0)
[junit4:junit4] Completed on J1 in 29.05s, 1 test, 2 errors <<< FAILURES!

[...truncated 476 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 814 ignored

Total time: 20 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 827 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/827/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestBinaryField

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.schema.TestBinaryField:     1) Thread[id=35, name=qtp943108539-35 Acceptor0 SelectChannelConnector@0.0.0.0:64024, state=RUNNABLE, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=37, name=qtp943108539-37 Acceptor2 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=36, name=qtp943108539-36 Acceptor1 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=38, name=qtp943108539-38 Acceptor3 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.schema.TestBinaryField: 
   1) Thread[id=35, name=qtp943108539-35 Acceptor0 SelectChannelConnector@0.0.0.0:64024, state=RUNNABLE, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=37, name=qtp943108539-37 Acceptor2 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=36, name=qtp943108539-36 Acceptor1 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=38, name=qtp943108539-38 Acceptor3 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([BC0299FA3AFA0575]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestBinaryField

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=35, name=qtp943108539-35 Acceptor0 SelectChannelConnector@0.0.0.0:64024, state=RUNNABLE, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=37, name=qtp943108539-37 Acceptor2 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=36, name=qtp943108539-36 Acceptor1 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=38, name=qtp943108539-38 Acceptor3 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=35, name=qtp943108539-35 Acceptor0 SelectChannelConnector@0.0.0.0:64024, state=RUNNABLE, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=37, name=qtp943108539-37 Acceptor2 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=36, name=qtp943108539-36 Acceptor1 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=38, name=qtp943108539-38 Acceptor3 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([BC0299FA3AFA0575]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.NoCacheHeaderTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest:     1) Thread[id=13, name=qtp2136814472-13 Acceptor0 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=15, name=qtp2136814472-15 Acceptor2 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=18, name=qtp2136814472-18 Acceptor3 SelectChannelConnector@0.0.0.0:24713, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=17, name=qtp2136814472-17 Acceptor1 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest: 
   1) Thread[id=13, name=qtp2136814472-13 Acceptor0 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=15, name=qtp2136814472-15 Acceptor2 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=18, name=qtp2136814472-18 Acceptor3 SelectChannelConnector@0.0.0.0:24713, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=17, name=qtp2136814472-17 Acceptor1 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([BC0299FA3AFA0575]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.NoCacheHeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=13, name=qtp2136814472-13 Acceptor0 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=15, name=qtp2136814472-15 Acceptor2 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=18, name=qtp2136814472-18 Acceptor3 SelectChannelConnector@0.0.0.0:24713, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=17, name=qtp2136814472-17 Acceptor1 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=13, name=qtp2136814472-13 Acceptor0 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=15, name=qtp2136814472-15 Acceptor2 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=18, name=qtp2136814472-18 Acceptor3 SelectChannelConnector@0.0.0.0:24713, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=17, name=qtp2136814472-17 Acceptor1 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([BC0299FA3AFA0575]:0)




Build Log:
[...truncated 7633 lines...]
[junit4:junit4] Suite: org.apache.solr.servlet.NoCacheHeaderTest
[junit4:junit4]   2> 173 T10 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-NoCacheHeaderTest-1351765855140
[junit4:junit4]   2> 178 T10 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 623 T10 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 900 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:24713
[junit4:junit4]   2> 1043 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1044 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1045 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1607 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1610 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1611 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1640 T10 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1644 T10 oasc.CoreContainer.<init> New CoreContainer 857966469
[junit4:junit4]   2> 1645 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1645 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1714 T10 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1791 T10 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1792 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1794 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1794 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1878 T10 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2011 T10 oasc.SolrConfig.<init> Loaded SolrConfig: solr/collection1/conf/solrconfig-nocache.xml
[junit4:junit4]   2> 2041 T10 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2103 T10 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 3801 T10 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 3828 T10 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 3832 T10 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 4132 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4144 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4149 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4165 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4246 T10 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351765855140/
[junit4:junit4]   2> 4247 T10 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 4294 T10 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351765855140/index/
[junit4:junit4]   2> 4295 T10 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351765855140/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 4381 T10 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351765855140/index forceNew:false
[junit4:junit4]   2> 4683 T10 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@5675b3ee lockFactory=org.apache.lucene.store.NativeFSLockFactory@495a2dc4; maxCacheMB=0.6123046875 maxMergeSizeMB=0.931640625)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4684 T10 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5456 T10 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5515 T10 oass.SolrIndexSearcher.<init> Opening Searcher@3c6b93c5 main
[junit4:junit4]   2> 5516 T10 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5575 T10 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5575 T10 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5666 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 5666 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 5667 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 5667 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 5668 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 5668 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 5669 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 5669 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 5670 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 5700 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 6098 T10 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 6099 T24 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3c6b93c5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6099 T10 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 6101 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 6101 T10 oas.SolrJettyTestBase.createJetty Jetty Assigned Port#24713
[junit4:junit4]   2> 6138 T10 oas.SolrTestCaseJ4.setUp ###Starting testCacheControl
[junit4:junit4]   2> 6160 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@73f5eb7f
[junit4:junit4]   2> 7821 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=454 
[junit4:junit4]   2> 8049 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8057 T23 C0 REQ [collection1] webapp=/solr path=/select params={q=solr&qt=standard} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 8058 T10 oas.SolrTestCaseJ4.tearDown ###Ending testCacheControl
[junit4:junit4]   2> 8350 T10 oas.SolrTestCaseJ4.setUp ###Starting testEtag
[junit4:junit4]   2> 8355 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8359 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8363 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 8368 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8372 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8376 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 8380 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8383 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 8386 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 8389 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 8390 T10 oas.SolrTestCaseJ4.tearDown ###Ending testEtag
[junit4:junit4]   2> 8404 T10 oas.SolrTestCaseJ4.setUp ###Starting testLastModified
[junit4:junit4]   2> 8408 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8430 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8434 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8438 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8442 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8446 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 8450 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8454 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8458 T22 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 8461 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 8462 T10 oas.SolrTestCaseJ4.tearDown ###Ending testLastModified
[junit4:junit4]   2> 8512 T10 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=857966469
[junit4:junit4]   2> 8513 T10 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@73f5eb7f
[junit4:junit4]   2> 8514 T10 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 8514 T10 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 8515 T10 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 8515 T10 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 8519 T10 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 8542 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 8644 T10 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 8644 T10 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 8697 T9 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 28794 T9 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest: 
[junit4:junit4]   2> 	   1) Thread[id=13, name=qtp2136814472-13 Acceptor0 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=15, name=qtp2136814472-15 Acceptor2 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=18, name=qtp2136814472-18 Acceptor3 SelectChannelConnector@0.0.0.0:24713, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=17, name=qtp2136814472-17 Acceptor1 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 28795 T9 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=13, name=qtp2136814472-13 Acceptor0 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	   2) Thread[id=15, name=qtp2136814472-15 Acceptor2 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	   3) Thread[id=18, name=qtp2136814472-18 Acceptor3 SelectChannelConnector@0.0.0.0:24713, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	   4) Thread[id=17, name=qtp2136814472-17 Acceptor1 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 31839 T9 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=13, name=qtp2136814472-13 Acceptor0 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=15, name=qtp2136814472-15 Acceptor2 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=18, name=qtp2136814472-18 Acceptor3 SelectChannelConnector@0.0.0.0:24713, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=17, name=qtp2136814472-17 Acceptor1 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {}, sim=DefaultSimilarity, locale=el, timezone=Africa/Niamey
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=124022408,total=167116800
[junit4:junit4]   2> NOTE: All tests run in this JVM: [NoCacheHeaderTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=NoCacheHeaderTest -Dtests.seed=BC0299FA3AFA0575 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=el -Dtests.timezone=Africa/Niamey -Dtests.file.encoding=ISO8859-1
[junit4:junit4] ERROR   0.00s J0 | NoCacheHeaderTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest: 
[junit4:junit4]    >    1) Thread[id=13, name=qtp2136814472-13 Acceptor0 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=15, name=qtp2136814472-15 Acceptor2 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=18, name=qtp2136814472-18 Acceptor3 SelectChannelConnector@0.0.0.0:24713, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=17, name=qtp2136814472-17 Acceptor1 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC0299FA3AFA0575]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=13, name=qtp2136814472-13 Acceptor0 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=15, name=qtp2136814472-15 Acceptor2 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=18, name=qtp2136814472-18 Acceptor3 SelectChannelConnector@0.0.0.0:24713, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=17, name=qtp2136814472-17 Acceptor1 SelectChannelConnector@0.0.0.0:24713, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC0299FA3AFA0575]:0)
[junit4:junit4] Completed on J0 in 33.58s, 3 tests, 2 errors <<< FAILURES!

[...truncated 256 lines...]
[junit4:junit4] Suite: org.apache.solr.schema.TestBinaryField
[junit4:junit4]   2> 1535 T28 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/collection1/'
[junit4:junit4]   2> 1914 T28 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2154 T28 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:64024
[junit4:junit4]   2> 2179 T28 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2179 T28 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example
[junit4:junit4]   2> 2180 T28 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/'
[junit4:junit4]   2> 2208 T28 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2209 T28 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2209 T28 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example
[junit4:junit4]   2> 2209 T28 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/solr.xml
[junit4:junit4]   2> 2210 T28 oasc.CoreContainer.<init> New CoreContainer 1927095247
[junit4:junit4]   2> 2210 T28 oasc.CoreContainer$Initializer.initialize no solr.xml file found - using default
[junit4:junit4]   2> 2211 T28 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/'
[junit4:junit4]   2> 2211 T28 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/'
[junit4:junit4]   2> 2249 T28 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2291 T28 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/collection1
[junit4:junit4]   2> 2292 T28 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/collection1/'
[junit4:junit4]   2> 2333 T28 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2388 T28 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2389 T28 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2393 T28 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2478 T28 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2480 T28 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/collection1/data/
[junit4:junit4]   2> 2480 T28 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 2481 T28 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/collection1/data/index/
[junit4:junit4]   2> 2482 T28 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/collection1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2486 T28 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestBinaryField-1351765870331/example/collection1/data/index forceNew:false
[junit4:junit4]   2> 2487 T28 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2fe6707 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3f2221f6),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2488 T28 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2489 T28 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2489 T28 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2490 T28 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2490 T28 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2490 T28 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2519 T28 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2524 T28 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2580 T28 oass.SolrIndexSearcher.<init> Opening Searcher@34f65b5b main
[junit4:junit4]   2> 2581 T28 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 2581 T28 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2582 T28 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2583 T28 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2583 T28 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2584 T28 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2585 T28 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2585 T28 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2586 T28 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2586 T28 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2587 T28 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2588 T28 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2588 T28 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2614 T28 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 2615 T39 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@34f65b5b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2615 T28 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2616 T28 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2616 T28 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2687 T28 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@24cc17f1
[junit4:junit4]   2> 3979 T43 C5 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2fe6707 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3f2221f6),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3980 T43 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3982 T43 C5 UPDATE [collection1] webapp=/example path=/update params={wt=javabin&version=2} {add=[1]} 0 41
[junit4:junit4]   2> 4161 T40 C5 UPDATE [collection1] webapp=/example path=/update params={wt=javabin&version=2} {add=[2]} 0 1
[junit4:junit4]   2> 4166 T43 C5 UPDATE [collection1] webapp=/example path=/update params={wt=javabin&version=2} {add=[3]} 0 0
[junit4:junit4]   2> 4200 T40 C5 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 4203 T40 C5 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2fe6707 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3f2221f6),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2fe6707 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3f2221f6),segFN=segments_2,generation=2,filenames=[_0.fnm, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.doc, _0.tip, _0.fdt]
[junit4:junit4]   2> 4204 T40 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 4207 T40 C5 oass.SolrIndexSearcher.<init> Opening Searcher@3a67ad79 main
[junit4:junit4]   2> 4208 T40 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4209 T40 C5 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 4209 T39 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a67ad79 main{StandardDirectoryReader(segments_2:3 _0(4.1):C3)}
[junit4:junit4]   2> 4210 T40 C5 UPDATE [collection1] webapp=/example path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 10
[junit4:junit4]   2> 4278 T42 C5 REQ [collection1] webapp=/example path=/select params={wt=javabin&q=*:*&version=2} hits=3 status=0 QTime=38 
[junit4:junit4]   2> 4342 T28 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1927095247
[junit4:junit4]   2> 4342 T28 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@24cc17f1
[junit4:junit4]   2> 4343 T28 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 4344 T28 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 4345 T28 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 4345 T28 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 4347 T28 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 4352 T28 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/example,null}
[junit4:junit4]   2> 4452 T28 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 4458 T27 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 24557 T27 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 4 threads leaked from SUITE scope at org.apache.solr.schema.TestBinaryField: 
[junit4:junit4]   2> 	   1) Thread[id=35, name=qtp943108539-35 Acceptor0 SelectChannelConnector@0.0.0.0:64024, state=RUNNABLE, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=37, name=qtp943108539-37 Acceptor2 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=36, name=qtp943108539-36 Acceptor1 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=38, name=qtp943108539-38 Acceptor3 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 24558 T27 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=35, name=qtp943108539-35 Acceptor0 SelectChannelConnector@0.0.0.0:64024, state=RUNNABLE, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	   2) Thread[id=37, name=qtp943108539-37 Acceptor2 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	   3) Thread[id=36, name=qtp943108539-36 Acceptor1 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	   4) Thread[id=38, name=qtp943108539-38 Acceptor3 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 27602 T27 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=35, name=qtp943108539-35 Acceptor0 SelectChannelConnector@0.0.0.0:64024, state=RUNNABLE, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=37, name=qtp943108539-37 Acceptor2 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=36, name=qtp943108539-36 Acceptor1 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=38, name=qtp943108539-38 Acceptor3 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ar_QA, timezone=Europe/Vienna
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=123682000,total=206307328
[junit4:junit4]   2> NOTE: All tests run in this JVM: [QueryParsingTest, TestReversedWildcardFilterFactory, TestArbitraryIndexDir, TestCollationKeyRangeQueries, FieldMutatingUpdateProcessorTest, TestFoldingMultitermQuery, TestBinaryField]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBinaryField -Dtests.seed=BC0299FA3AFA0575 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_QA -Dtests.timezone=Europe/Vienna -Dtests.file.encoding=ISO8859-1
[junit4:junit4] ERROR   0.00s J1 | TestBinaryField (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.schema.TestBinaryField: 
[junit4:junit4]    >    1) Thread[id=35, name=qtp943108539-35 Acceptor0 SelectChannelConnector@0.0.0.0:64024, state=RUNNABLE, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=37, name=qtp943108539-37 Acceptor2 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=36, name=qtp943108539-36 Acceptor1 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=38, name=qtp943108539-38 Acceptor3 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC0299FA3AFA0575]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=35, name=qtp943108539-35 Acceptor0 SelectChannelConnector@0.0.0.0:64024, state=RUNNABLE, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=37, name=qtp943108539-37 Acceptor2 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=36, name=qtp943108539-36 Acceptor1 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=38, name=qtp943108539-38 Acceptor3 SelectChannelConnector@0.0.0.0:64024, state=BLOCKED, group=TGRP-TestBinaryField]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC0299FA3AFA0575]:0)
[junit4:junit4] Completed on J1 in 26.23s, 1 test, 2 errors <<< FAILURES!

[...truncated 431 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 822 ignored

Total time: 21 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 826 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/826/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=66, name=qtp1731015846-66 Acceptor2 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=97, name=qtp1861758951-97 Acceptor1 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=96, name=qtp1861758951-96 Acceptor0 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=116, name=qtp1177842774-116 Acceptor2 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=48, name=qtp1149059701-48 Acceptor2 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=98, name=qtp1861758951-98 Acceptor2 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=114, name=qtp1177842774-114 Acceptor0 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=64, name=qtp1731015846-64 Acceptor0 SelectChannelConnector@0.0.0.0:12818, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=67, name=qtp1731015846-67 Acceptor3 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=49, name=qtp1149059701-49 Acceptor3 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=115, name=qtp1177842774-115 Acceptor1 SelectChannelConnector@0.0.0.0:56173, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=47, name=qtp1149059701-47 Acceptor1 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=65, name=qtp1731015846-65 Acceptor1 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=117, name=qtp1177842774-117 Acceptor3 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=46, name=qtp1149059701-46 Acceptor0 SelectChannelConnector@0.0.0.0:34217, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=99, name=qtp1861758951-99 Acceptor3 SelectChannelConnector@0.0.0.0:54385, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=66, name=qtp1731015846-66 Acceptor2 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=97, name=qtp1861758951-97 Acceptor1 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=96, name=qtp1861758951-96 Acceptor0 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=116, name=qtp1177842774-116 Acceptor2 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=48, name=qtp1149059701-48 Acceptor2 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=98, name=qtp1861758951-98 Acceptor2 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=114, name=qtp1177842774-114 Acceptor0 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=64, name=qtp1731015846-64 Acceptor0 SelectChannelConnector@0.0.0.0:12818, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=67, name=qtp1731015846-67 Acceptor3 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=49, name=qtp1149059701-49 Acceptor3 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=115, name=qtp1177842774-115 Acceptor1 SelectChannelConnector@0.0.0.0:56173, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=47, name=qtp1149059701-47 Acceptor1 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=65, name=qtp1731015846-65 Acceptor1 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=117, name=qtp1177842774-117 Acceptor3 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=46, name=qtp1149059701-46 Acceptor0 SelectChannelConnector@0.0.0.0:34217, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=99, name=qtp1861758951-99 Acceptor3 SelectChannelConnector@0.0.0.0:54385, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([AE8CEDF37E49E507]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=66, name=qtp1731015846-66 Acceptor2 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=96, name=qtp1861758951-96 Acceptor0 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=97, name=qtp1861758951-97 Acceptor1 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=116, name=qtp1177842774-116 Acceptor2 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=48, name=qtp1149059701-48 Acceptor2 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=98, name=qtp1861758951-98 Acceptor2 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=64, name=qtp1731015846-64 Acceptor0 SelectChannelConnector@0.0.0.0:12818, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=114, name=qtp1177842774-114 Acceptor0 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=67, name=qtp1731015846-67 Acceptor3 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=49, name=qtp1149059701-49 Acceptor3 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=115, name=qtp1177842774-115 Acceptor1 SelectChannelConnector@0.0.0.0:56173, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=65, name=qtp1731015846-65 Acceptor1 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=47, name=qtp1149059701-47 Acceptor1 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=117, name=qtp1177842774-117 Acceptor3 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=46, name=qtp1149059701-46 Acceptor0 SelectChannelConnector@0.0.0.0:34217, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=99, name=qtp1861758951-99 Acceptor3 SelectChannelConnector@0.0.0.0:54385, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=66, name=qtp1731015846-66 Acceptor2 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=96, name=qtp1861758951-96 Acceptor0 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=97, name=qtp1861758951-97 Acceptor1 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=116, name=qtp1177842774-116 Acceptor2 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=48, name=qtp1149059701-48 Acceptor2 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=98, name=qtp1861758951-98 Acceptor2 SelectChannelConnector@0.0.0.0:54385, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=64, name=qtp1731015846-64 Acceptor0 SelectChannelConnector@0.0.0.0:12818, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=114, name=qtp1177842774-114 Acceptor0 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=67, name=qtp1731015846-67 Acceptor3 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=49, name=qtp1149059701-49 Acceptor3 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=115, name=qtp1177842774-115 Acceptor1 SelectChannelConnector@0.0.0.0:56173, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=65, name=qtp1731015846-65 Acceptor1 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=47, name=qtp1149059701-47 Acceptor1 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=117, name=qtp1177842774-117 Acceptor3 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=46, name=qtp1149059701-46 Acceptor0 SelectChannelConnector@0.0.0.0:34217, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=99, name=qtp1861758951-99 Acceptor3 SelectChannelConnector@0.0.0.0:54385, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([AE8CEDF37E49E507]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.PeerSyncTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.update.PeerSyncTest:     1) Thread[id=111, name=qtp1193945680-111 Acceptor3 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=96, name=qtp619193260-96 Acceptor2 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=109, name=qtp1193945680-109 Acceptor1 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=97, name=qtp619193260-97 Acceptor3 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=108, name=qtp1193945680-108 Acceptor0 SelectChannelConnector@0.0.0.0:10970, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=110, name=qtp1193945680-110 Acceptor2 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=80, name=qtp906842278-80 Acceptor0 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=69, name=qtp1095568998-69 Acceptor3 SelectChannelConnector@0.0.0.0:17753, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=67, name=qtp1095568998-67 Acceptor1 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=94, name=qtp619193260-94 Acceptor0 SelectChannelConnector@0.0.0.0:63030, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=68, name=qtp1095568998-68 Acceptor2 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=95, name=qtp619193260-95 Acceptor1 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=82, name=qtp906842278-82 Acceptor2 SelectChannelConnector@0.0.0.0:16794, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=83, name=qtp906842278-83 Acceptor3 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=66, name=qtp1095568998-66 Acceptor0 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=81, name=qtp906842278-81 Acceptor1 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.update.PeerSyncTest: 
   1) Thread[id=111, name=qtp1193945680-111 Acceptor3 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=96, name=qtp619193260-96 Acceptor2 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=109, name=qtp1193945680-109 Acceptor1 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=97, name=qtp619193260-97 Acceptor3 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=108, name=qtp1193945680-108 Acceptor0 SelectChannelConnector@0.0.0.0:10970, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=110, name=qtp1193945680-110 Acceptor2 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=80, name=qtp906842278-80 Acceptor0 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=69, name=qtp1095568998-69 Acceptor3 SelectChannelConnector@0.0.0.0:17753, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=67, name=qtp1095568998-67 Acceptor1 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=94, name=qtp619193260-94 Acceptor0 SelectChannelConnector@0.0.0.0:63030, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=68, name=qtp1095568998-68 Acceptor2 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=95, name=qtp619193260-95 Acceptor1 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=82, name=qtp906842278-82 Acceptor2 SelectChannelConnector@0.0.0.0:16794, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=83, name=qtp906842278-83 Acceptor3 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=66, name=qtp1095568998-66 Acceptor0 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=81, name=qtp906842278-81 Acceptor1 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([AE8CEDF37E49E507]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.PeerSyncTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=96, name=qtp619193260-96 Acceptor2 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=111, name=qtp1193945680-111 Acceptor3 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=109, name=qtp1193945680-109 Acceptor1 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=97, name=qtp619193260-97 Acceptor3 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=108, name=qtp1193945680-108 Acceptor0 SelectChannelConnector@0.0.0.0:10970, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=110, name=qtp1193945680-110 Acceptor2 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=80, name=qtp906842278-80 Acceptor0 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=67, name=qtp1095568998-67 Acceptor1 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=69, name=qtp1095568998-69 Acceptor3 SelectChannelConnector@0.0.0.0:17753, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=94, name=qtp619193260-94 Acceptor0 SelectChannelConnector@0.0.0.0:63030, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=68, name=qtp1095568998-68 Acceptor2 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=95, name=qtp619193260-95 Acceptor1 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=82, name=qtp906842278-82 Acceptor2 SelectChannelConnector@0.0.0.0:16794, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=83, name=qtp906842278-83 Acceptor3 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=66, name=qtp1095568998-66 Acceptor0 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=81, name=qtp906842278-81 Acceptor1 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=96, name=qtp619193260-96 Acceptor2 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=111, name=qtp1193945680-111 Acceptor3 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=109, name=qtp1193945680-109 Acceptor1 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=97, name=qtp619193260-97 Acceptor3 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=108, name=qtp1193945680-108 Acceptor0 SelectChannelConnector@0.0.0.0:10970, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=110, name=qtp1193945680-110 Acceptor2 SelectChannelConnector@0.0.0.0:10970, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=80, name=qtp906842278-80 Acceptor0 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=67, name=qtp1095568998-67 Acceptor1 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=69, name=qtp1095568998-69 Acceptor3 SelectChannelConnector@0.0.0.0:17753, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=94, name=qtp619193260-94 Acceptor0 SelectChannelConnector@0.0.0.0:63030, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=68, name=qtp1095568998-68 Acceptor2 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=95, name=qtp619193260-95 Acceptor1 SelectChannelConnector@0.0.0.0:63030, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=82, name=qtp906842278-82 Acceptor2 SelectChannelConnector@0.0.0.0:16794, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=83, name=qtp906842278-83 Acceptor3 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=66, name=qtp1095568998-66 Acceptor0 SelectChannelConnector@0.0.0.0:17753, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=81, name=qtp906842278-81 Acceptor1 SelectChannelConnector@0.0.0.0:16794, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([AE8CEDF37E49E507]:0)




Build Log:
[...truncated 7655 lines...]
[junit4:junit4] Suite: org.apache.solr.update.PeerSyncTest
[junit4:junit4]   2> 4 T59 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 628 T59 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 915 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:17753
[junit4:junit4]   2> 940 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 941 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 941 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 970 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 971 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 972 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 972 T59 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 972 T59 oasc.CoreContainer.<init> New CoreContainer 388368796
[junit4:junit4]   2> 973 T59 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 974 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1007 T59 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1044 T59 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1045 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1047 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1048 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1082 T59 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1118 T59 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 1119 T59 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1127 T59 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1563 T59 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1572 T59 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1575 T59 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1586 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1590 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1593 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1594 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1595 T59 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351743821759/control/data/
[junit4:junit4]   2> 1596 T59 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 1596 T59 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351743821759/control/data/index/
[junit4:junit4]   2> 1597 T59 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351743821759/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1600 T59 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351743821759/control/data/index forceNew:false
[junit4:junit4]   2> 1603 T59 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.MMapDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/index7235772358tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@29952be0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1604 T59 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1610 T59 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1610 T59 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1611 T59 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1611 T59 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1616 T59 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1622 T59 oass.SolrIndexSearcher.<init> Opening Searcher@670a59d2 main
[junit4:junit4]   2> 1623 T59 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1645 T59 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1645 T59 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1646 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1646 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1647 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1647 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1647 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1648 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1648 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1649 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1649 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1649 T59 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1660 T74 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@670a59d2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1663 T59 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1664 T59 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 1664 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1711 T59 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1727 T59 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1729 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:16794
[junit4:junit4]   2> 1730 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1730 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1730 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1754 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1754 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1755 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1755 T59 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1755 T59 oasc.CoreContainer.<init> New CoreContainer 325174786
[junit4:junit4]   2> 1756 T59 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1756 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1774 T59 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1787 T59 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1788 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1789 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1790 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1820 T59 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1856 T59 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 1857 T59 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1866 T59 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2277 T59 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2285 T59 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2288 T59 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2298 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2302 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2305 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2306 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2307 T59 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351743821759/shard0/data/
[junit4:junit4]   2> 2307 T59 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 2308 T59 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351743821759/shard0/data/index/
[junit4:junit4]   2> 2309 T59 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351743821759/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2310 T59 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351743821759/shard0/data/index forceNew:false
[junit4:junit4]   2> 2313 T59 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2313 T59 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2314 T59 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2315 T59 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2315 T59 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2315 T59 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2320 T59 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2325 T59 oass.SolrIndexSearcher.<init> Opening Searcher@441e037a main
[junit4:junit4]   2> 2325 T59 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 2326 T59 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2326 T59 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2327 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2327 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2327 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2328 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2328 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2329 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2329 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2329 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2330 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2330 T59 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2340 T59 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2340 T88 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@441e037a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2340 T59 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2341 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2341 T59 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2349 T59 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2351 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:63030
[junit4:junit4]   2> 2352 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2352 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2353 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2374 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2375 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2375 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2375 T59 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2376 T59 oasc.CoreContainer.<init> New CoreContainer 189300272
[junit4:junit4]   2> 2376 T59 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2377 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2394 T59 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2407 T59 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2408 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2409 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2409 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2438 T59 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2473 T59 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 2473 T59 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2481 T59 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2891 T59 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2899 T59 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2902 T59 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2912 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2916 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2919 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2920 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2921 T59 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351743821759/shard1/data/
[junit4:junit4]   2> 2921 T59 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 2922 T59 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351743821759/shard1/data/index/
[junit4:junit4]   2> 2923 T59 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351743821759/shard1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2924 T59 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351743821759/shard1/data/index forceNew:false
[junit4:junit4]   2> 2926 T59 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2927 T59 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2928 T59 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2928 T59 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2929 T59 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2929 T59 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2934 T59 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2938 T59 oass.SolrIndexSearcher.<init> Opening Searcher@81ee8c1 main
[junit4:junit4]   2> 2938 T59 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 2939 T59 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2940 T59 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2940 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2940 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2941 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2941 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2941 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2942 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2942 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2942 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2943 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2943 T59 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2953 T59 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2953 T102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@81ee8c1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2954 T59 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2954 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2955 T59 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2962 T59 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2964 T59 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:10970
[junit4:junit4]   2> 2965 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2966 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2966 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2988 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2988 T59 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2988 T59 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2989 T59 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2989 T59 oasc.CoreContainer.<init> New CoreContainer 1853698227
[junit4:junit4]   2> 2990 T59 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2990 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3007 T59 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3020 T59 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 3021 T59 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 3022 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 3023 T59 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 3052 T59 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 3086 T59 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 3087 T59 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 3095 T59 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 3513 T59 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 3522 T59 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 3525 T59 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 3535 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3539 T59 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3542 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3543 T59 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3544 T59 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351743821759/shard2/data/
[junit4:junit4]   2> 3544 T59 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 3545 T59 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351743821759/shard2/data/index/
[junit4:junit4]   2> 3546 T59 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351743821759/shard2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 3547 T59 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351743821759/shard2/data/index forceNew:false
[junit4:junit4]   2> 3549 T59 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5d73b35d lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c97cb70),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3550 T59 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3551 T59 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 3551 T59 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 3551 T59 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 3552 T59 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 3557 T59 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 3561 T59 oass.SolrIndexSearcher.<init> Opening Searcher@70991d56 main
[junit4:junit4]   2> 3561 T59 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3562 T59 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 3563 T59 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 3563 T59 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 3563 T59 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3564 T59 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 3564 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3564 T59 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3565 T59 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3565 T59 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3566 T59 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3566 T59 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3566 T59 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 3576 T116 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@70991d56 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 3576 T59 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 3577 T59 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3577 T59 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3577 T59 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@6a46ca65
[junit4:junit4]   2> 4559 T85 C6 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4560 T85 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4593 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=63 
[junit4:junit4]   2> ASYNC  NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@350204ce
[junit4:junit4]   2> 4822 T100 C7 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 4846 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:16794/solr] nUpdates=100
[junit4:junit4]   2> 4857 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 4858 T100 C7 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:16794/solr&version=2&getVersions=100} status=0 QTime=102 
[junit4:junit4]   2> 4869 T99 C7 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4869 T99 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4881 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=16 
[junit4:junit4]   2> 4886 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:16794/solr] nUpdates=100
[junit4:junit4]   2> 4896 T87 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=13 
[junit4:junit4]   2> 4896 T86 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=8 
[junit4:junit4]   2> 4897 T100 C7 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 1 versions from 127.0.0.1:16794/solr
[junit4:junit4]   2> 4898 T100 C7 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Our versions are newer. ourLowThreshold=1 otherHigh=1
[junit4:junit4]   2> 4898 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 4899 T100 C7 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:16794/solr&version=2&getVersions=100} status=0 QTime=14 
[junit4:junit4]   2> 4908 T85 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 4911 T85 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_2,generation=2,filenames=[_0_1.len, _0_3.len, _0_5.len, _0_6.len, segments_2, _0.inf, _0.si, _0.pst, _0.fld]
[junit4:junit4]   2> 4911 T85 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 4919 T85 C6 oass.SolrIndexSearcher.<init> Opening Searcher@5999ef99 main
[junit4:junit4]   2> 4919 T85 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 4929 T88 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5999ef99 main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 4929 T85 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 4930 T85 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 22
[junit4:junit4]   2> 4934 T101 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 4937 T101 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_2,generation=2,filenames=[_0_1.len, _0_3.len, _0_5.len, _0_6.len, segments_2, _0.inf, _0.si, _0.pst, _0.fld]
[junit4:junit4]   2> 4938 T101 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 4941 T101 C7 oass.SolrIndexSearcher.<init> Opening Searcher@677bc0ac main
[junit4:junit4]   2> 4941 T101 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4942 T102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@677bc0ac main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 4942 T101 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 4943 T101 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 9
[junit4:junit4]   2> 4965 T87 C6 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=1 status=0 QTime=2 
[junit4:junit4]   2> 4974 T99 C7 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 4990 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 4994 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:16794/solr] nUpdates=100
[junit4:junit4]   2> 4998 T86 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=0 
[junit4:junit4]   2> 4999 T100 C7 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 2 versions from 127.0.0.1:16794/solr
[junit4:junit4]   2> 4999 T100 C7 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:16794/solrn=1 versions=[2]
[junit4:junit4]   2> 5003 T85 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=2&qt=/get&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 5009 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 5009 T100 C7 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:16794/solr&version=2&getVersions=100} status=0 QTime=16 
[junit4:junit4]   2> 5013 T87 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5021 T87 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_2,generation=2,filenames=[_0_1.len, _0_3.len, _0_5.len, _0_6.len, segments_2, _0.inf, _0.si, _0.pst, _0.fld]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_3,generation=3,filenames=[_1_1.len, _1.fld, _0_1.len, _0_6.len, _0_5.len, _1.inf, _1_15.len, _0.pst, _0.fld, _1.pst, _0_3.len, _1_3.len, _1_6.len, _1.si, _0.inf, _0.si, _1_5.len, segments_3, _1_16.len]
[junit4:junit4]   2> 5021 T87 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 5026 T87 C6 oass.SolrIndexSearcher.<init> Opening Searcher@f7bd29 main
[junit4:junit4]   2> 5026 T87 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 5027 T88 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f7bd29 main{StandardDirectoryReader(segments_3:5 _0(4.1):C1 _1(4.1):C1)}
[junit4:junit4]   2> 5028 T87 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5028 T87 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 15
[junit4:junit4]   2> 5034 T101 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5041 T101 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_2,generation=2,filenames=[_0_1.len, _0_3.len, _0_5.len, _0_6.len, segments_2, _0.inf, _0.si, _0.pst, _0.fld]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_3,generation=3,filenames=[_1_1.len, _1.fld, _0_1.len, _0_6.len, _0_5.len, _1.inf, _1_15.len, _0.pst, _0.fld, _1.pst, _0_3.len, _1_3.len, _1_6.len, _1.si, _0.inf, _0.si, _1_5.len, segments_3, _1_16.len]
[junit4:junit4]   2> 5041 T101 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 5045 T101 C7 oass.SolrIndexSearcher.<init> Opening Searcher@321893c7 main
[junit4:junit4]   2> 5045 T101 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5049 T102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@321893c7 main{StandardDirectoryReader(segments_3:5 _0(4.1):C1 _1(4.1):C1)}
[junit4:junit4]   2> 5049 T101 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5050 T101 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 16
[junit4:junit4]   2> 5054 T86 C6 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 5060 T99 C7 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 5070 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 5079 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 5088 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 5095 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 5103 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 5110 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 5117 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 5124 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 5128 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:16794/solr] nUpdates=100
[junit4:junit4]   2> 5132 T86 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=0 
[junit4:junit4]   2> 5133 T100 C7 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 10 versions from 127.0.0.1:16794/solr
[junit4:junit4]   2> 5133 T100 C7 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:16794/solrn=8 versions=[10, 9, 8, 7, 6, 5, 4, 3]
[junit4:junit4]   2> 5138 T85 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=10,9,8,7,6,5,4,3&qt=/get&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5165 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 5165 T100 C7 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:16794/solr&version=2&getVersions=100} status=0 QTime=38 
[junit4:junit4]   2> 5169 T87 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5180 T87 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_3,generation=3,filenames=[_1_1.len, _1.fld, _0_1.len, _0_6.len, _0_5.len, _1.inf, _1_15.len, _0.pst, _0.fld, _1.pst, _0_3.len, _1_3.len, _1_6.len, _1.si, _0.inf, _0.si, _1_5.len, segments_3, _1_16.len]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_4,generation=4,filenames=[_1_1.len, _2.si, _1.fld, _0_1.len, _2.inf, _1.inf, _2_6.len, _1.pst, _2_5.len, _0.si, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _2_3.len, _2_1.len, _0.pst, _1_15.len, _0.fld, _2.pst, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _0.inf, segments_4]
[junit4:junit4]   2> 5181 T87 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 5184 T87 C6 oass.SolrIndexSearcher.<init> Opening Searcher@458db581 main
[junit4:junit4]   2> 5184 T87 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 5185 T87 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5185 T88 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@458db581 main{StandardDirectoryReader(segments_4:7 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8)}
[junit4:junit4]   2> 5186 T87 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 17
[junit4:junit4]   2> 5190 T99 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 5206 T99 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_3,generation=3,filenames=[_1_1.len, _1.fld, _0_1.len, _0_6.len, _0_5.len, _1.inf, _1_15.len, _0.pst, _0.fld, _1.pst, _0_3.len, _1_3.len, _1_6.len, _1.si, _0.inf, _0.si, _1_5.len, segments_3, _1_16.len]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_4,generation=4,filenames=[_1_1.len, _2.si, _1.fld, _0_1.len, _2.inf, _1.inf, _2_6.len, _1.pst, _2_5.len, _0.si, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _2_3.len, _2_1.len, _0.pst, _1_15.len, _0.fld, _2.pst, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _0.inf, segments_4]
[junit4:junit4]   2> 5207 T99 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 5211 T99 C7 oass.SolrIndexSearcher.<init> Opening Searcher@4fbf7435 main
[junit4:junit4]   2> 5211 T99 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5212 T102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4fbf7435 main{StandardDirectoryReader(segments_4:7 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8)}
[junit4:junit4]   2> 5212 T99 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 5213 T99 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 23
[junit4:junit4]   2> 5217 T86 C6 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=10 status=0 QTime=1 
[junit4:junit4]   2> 5226 T101 C7 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=10 status=0 QTime=1 
[junit4:junit4]   2> 5242 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 5248 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5253 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5258 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5263 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5269 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5274 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5279 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5284 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5289 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5294 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5299 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5304 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5309 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5314 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5319 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5324 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5329 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5335 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5340 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5345 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5351 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5356 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5362 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5367 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5373 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5379 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5385 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5390 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5395 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5400 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5406 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5411 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5416 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5421 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5426 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5431 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5436 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5443 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5449 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5456 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5462 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5469 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5474 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5479 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5484 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5489 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5494 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5499 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5504 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5509 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5514 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5519 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5524 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5529 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5534 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5540 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5545 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5550 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5555 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5560 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5565 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5572 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5577 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5582 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5587 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5592 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5598 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5603 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5609 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5614 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5620 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5625 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5632 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 5638 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5643 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5648 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5654 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5659 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5664 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5670 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5676 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5682 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5687 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5692 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5697 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5702 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5708 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5714 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5719 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5724 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5730 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5735 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5740 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5745 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5749 T101 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:16794/solr] nUpdates=100
[junit4:junit4]   2> 5754 T85 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 5755 T101 C7 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:16794/solr
[junit4:junit4]   2> 5756 T101 C7 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Our versions are too old. ourHighThreshold=8 otherLowThreshold=25
[junit4:junit4]   2> 5756 T101 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync failed
[junit4:junit4]   2> 5757 T101 C7 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:16794/solr&version=2&getVersions=100} status=0 QTime=9 
[junit4:junit4]   2> 5764 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 5770 T100 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5776 T101 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5783 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5789 T100 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5794 T101 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5799 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5804 T100 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5810 T101 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5817 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5822 T100 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5827 T101 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5832 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5837 T100 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5842 T101 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5847 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5852 T100 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5857 T101 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5862 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 5867 T100 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5872 T101 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5877 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5882 T100 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5887 T101 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5892 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 5896 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:16794/solr] nUpdates=100
[junit4:junit4]   2> 5901 T87 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 5902 T100 C7 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:16794/solr
[junit4:junit4]   2> 5903 T100 C7 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:16794/solrn=70 versions=[105, 104, 103, 102, 101, 100, 99, 98, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 79, 78, 77, 76, 75, 74, 73, 72, 71, 70, 69, 68, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 37, 36]
[junit4:junit4]   2> 5913 T86 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=105,104,103,102,101,100,99,98,97,96,95,94,93,92,91,90,89,88,87,86,85,84,83,82,81,80,79,78,77,76,75,74,73,72,71,70,69,68,67,66,65,64,63,62,61,60,59,58,57,56,55,54,53,52,51,50,49,48,47,46,45,44,43,42,41,40,39,38,37,36&qt=/get&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 5985 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 5985 T100 C7 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:16794/solr&version=2&getVersions=100} status=0 QTime=90 
[junit4:junit4]   2> 5989 T85 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6025 T85 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_4,generation=4,filenames=[_1_1.len, _2.si, _1.fld, _0_1.len, _2.inf, _1.inf, _2_6.len, _1.pst, _2_5.len, _0.si, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _2_3.len, _2_1.len, _0.pst, _1_15.len, _0.fld, _2.pst, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _0.inf, segments_4]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_5,generation=5,filenames=[_3.fld, _1_1.len, _2.si, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _1.inf, _2_6.len, _3.pst, _1.pst, _2_5.len, _0.si, _3_3.len, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _2_3.len, _2_1.len, _0.pst, _1_15.len, _0.fld, _3_1.len, _2.pst, _3.inf, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _3.si, _0.inf, segments_5]
[junit4:junit4]   2> 6026 T85 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 5
[junit4:junit4]   2> 6030 T85 C6 oass.SolrIndexSearcher.<init> Opening Searcher@b1afdf4 main
[junit4:junit4]   2> 6031 T85 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 6032 T88 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b1afdf4 main{StandardDirectoryReader(segments_5:9 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95)}
[junit4:junit4]   2> 6033 T85 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6033 T85 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 44
[junit4:junit4]   2> 6036 T99 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6063 T99 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_4,generation=4,filenames=[_1_1.len, _2.si, _1.fld, _0_1.len, _2.inf, _1.inf, _2_6.len, _1.pst, _2_5.len, _0.si, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _2_3.len, _2_1.len, _0.pst, _1_15.len, _0.fld, _2.pst, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _0.inf, segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_5,generation=5,filenames=[_3.fld, _1_1.len, _2.si, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _1.inf, _2_6.len, _3.pst, _1.pst, _2_5.len, _0.si, _3_3.len, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _2_3.len, _2_1.len, _0.pst, _1_15.len, _0.fld, _3_1.len, _2.pst, _3.inf, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _3.si, _0.inf, segments_5]
[junit4:junit4]   2> 6063 T99 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 5
[junit4:junit4]   2> 6070 T99 C7 oass.SolrIndexSearcher.<init> Opening Searcher@43877c42 main
[junit4:junit4]   2> 6070 T99 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6071 T102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43877c42 main{StandardDirectoryReader(segments_5:9 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95)}
[junit4:junit4]   2> 6072 T99 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6073 T99 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 37
[junit4:junit4]   2> 6089 T87 C6 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=105 status=0 QTime=13 
[junit4:junit4]   2> 6107 T100 C7 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=105 status=0 QTime=13 
[junit4:junit4]   2> 6122 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=8 
[junit4:junit4]   2> 6127 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6248 T87 C6 oass.SolrIndexSearcher.<init> Opening Searcher@402d659 realtime
[junit4:junit4]   2> 6248 T87 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 6249 T87 C6 REQ [collection1] webapp=/solr path=/update params={_version_=-1003&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=117 
[junit4:junit4]   2> 6257 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6263 T85 C6 REQ [collection1] webapp=/solr path=/update params={_version_=-1005&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6267 T101 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:16794/solr] nUpdates=100
[junit4:junit4]   2> 6275 T87 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=5 
[junit4:junit4]   2> 6277 T101 C7 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:16794/solr
[junit4:junit4]   2> 6277 T101 C7 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:16794/solrn=5 versions=[-1005, 1004, -1003, 1002, 1001]
[junit4:junit4]   2> 6282 T85 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=-1005,1004,-1003,1002,1001&qt=/get&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6313 T101 C7 oass.SolrIndexSearcher.<init> Opening Searcher@56a57bb2 realtime
[junit4:junit4]   2> 6314 T101 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6318 T101 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 6318 T101 C7 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:16794/solr&version=2&getVersions=100} status=0 QTime=52 
[junit4:junit4]   2> 6323 T86 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6327 T86 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_5,generation=5,filenames=[_3.fld, _1_1.len, _2.si, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _1.inf, _2_6.len, _3.pst, _1.pst, _2_5.len, _0.si, _3_3.len, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _2_3.len, _2_1.len, _0.pst, _1_15.len, _0.fld, _3_1.len, _2.pst, _3.inf, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _3.si, _0.inf, segments_5]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_6,generation=6,filenames=[_3.fld, _1_1.len, _2.si, _5_6.len, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _1.inf, _2_6.len, _3.pst, _1.pst, _2_5.len, _5_1.len, _5_3.len, _0.si, _3_3.len, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _5.pst, _2_3.len, _5.fld, _2_1.len, _0.pst, _1_15.len, _0.fld, _3_1.len, _5.si, _2.pst, _3.inf, _5.inf, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _3.si, _0.inf, segments_6, _5_5.len]
[junit4:junit4]   2> 6328 T86 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 6
[junit4:junit4]   2> 6330 T86 C6 oass.SolrIndexSearcher.<init> Opening Searcher@606f6a6a main
[junit4:junit4]   2> 6330 T86 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 6331 T88 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@606f6a6a main{StandardDirectoryReader(segments_6:16:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1)}
[junit4:junit4]   2> 6332 T86 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6334 T86 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 11
[junit4:junit4]   2> 6338 T99 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6343 T99 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_5,generation=5,filenames=[_3.fld, _1_1.len, _2.si, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _1.inf, _2_6.len, _3.pst, _1.pst, _2_5.len, _0.si, _3_3.len, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _2_3.len, _2_1.len, _0.pst, _1_15.len, _0.fld, _3_1.len, _2.pst, _3.inf, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _3.si, _0.inf, segments_5]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_6,generation=6,filenames=[_3.fld, _1_1.len, _2.si, _5_6.len, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _1.inf, _2_6.len, _3.pst, _1.pst, _2_5.len, _5_1.len, _5_3.len, _0.si, _3_3.len, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _5.pst, _2_3.len, _5.fld, _2_1.len, _0.pst, _1_15.len, _0.fld, _3_1.len, _5.si, _2.pst, _3.inf, _5.inf, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _3.si, _0.inf, segments_6, _5_5.len]
[junit4:junit4]   2> 6344 T99 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 6
[junit4:junit4]   2> 6346 T99 C7 oass.SolrIndexSearcher.<init> Opening Searcher@2f86780d main
[junit4:junit4]   2> 6347 T99 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6348 T102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2f86780d main{StandardDirectoryReader(segments_6:16:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1)}
[junit4:junit4]   2> 6349 T99 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6349 T99 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 11
[junit4:junit4]   2> 6360 T87 C6 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=106 status=0 QTime=8 
[junit4:junit4]   2> 6370 T100 C7 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=106 status=0 QTime=5 
[junit4:junit4]   2> 6378 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6383 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6393 T87 C6 oass.SolrIndexSearcher.<init> Opening Searcher@3cfe8153 realtime
[junit4:junit4]   2> 6394 T87 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 6395 T87 C6 REQ [collection1] webapp=/solr path=/update params={_version_=-2003&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=7 
[junit4:junit4]   2> 6400 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6404 T86 C6 REQ [collection1] webapp=/solr path=/update params={_version_=-2005&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6409 T99 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6421 T101 C7 oass.SolrIndexSearcher.<init> Opening Searcher@5561091e realtime
[junit4:junit4]   2> 6422 T101 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6423 T101 C7 REQ [collection1] webapp=/solr path=/update params={_version_=-2003&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=10 
[junit4:junit4]   2> 6428 T100 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6432 T99 C7 REQ [collection1] webapp=/solr path=/update params={_version_=-2005&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6436 T101 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:16794/solr] nUpdates=100
[junit4:junit4]   2> 6440 T87 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 6442 T101 C7 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:16794/solr
[junit4:junit4]   2> 6443 T101 C7 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:16794/solrn=1 versions=[2002]
[junit4:junit4]   2> 6446 T85 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=2002&qt=/get&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6448 T101 C7 oasu.DirectUpdateHandler2.addDoc Reordered DBQs detected.  Update=add{flags=12,_version_=2002,id=2001} DBQs=[DBQ{version=2003,q=id:2001 OR id:2002}]
[junit4:junit4]   2> 6460 T101 C7 oass.SolrIndexSearcher.<init> Opening Searcher@5b46c2cb realtime
[junit4:junit4]   2> 6461 T101 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6473 T101 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 6474 T101 C7 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:16794/solr&version=2&getVersions=100} status=0 QTime=39 
[junit4:junit4]   2> 6478 T86 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6482 T86 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_6,generation=6,filenames=[_3.fld, _1_1.len, _2.si, _5_6.len, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _1.inf, _2_6.len, _3.pst, _1.pst, _2_5.len, _5_1.len, _5_3.len, _0.si, _3_3.len, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _5.pst, _2_3.len, _5.fld, _2_1.len, _0.pst, _1_15.len, _0.fld, _3_1.len, _5.si, _2.pst, _3.inf, _5.inf, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _3.si, _0.inf, segments_6, _5_5.len]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_7,generation=7,filenames=[_1_1.len, _3.fld, _5_6.len, _7.si, _1.inf, _7_1.len, _2_6.len, _5_1.len, _1_5.len, _2.fld, _7.inf, _0_6.len, _0_5.len, _2_3.len, _5.fld, _5.si, _5.inf, _0_3.len, _1_6.len, _2_15.len, _0.inf, _3.si, _2.si, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _3.pst, _7_6.len, _1.pst, _2_5.len, _7.fld, _5_3.len, _0.si, _3_3.len, _1_16.len, _2_16.len, _7.pst, _5.pst, _1_15.len, _0.pst, _2_1.len, _0.fld, _3_1.len, _2.pst, _3.inf, _7_3.len, segments_7, _1_3.len, _1.si, _7_5.len, _5_5.len]
[junit4:junit4]   2> 6482 T86 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 7
[junit4:junit4]   2> 6485 T86 C6 oass.SolrIndexSearcher.<init> Opening Searcher@20f5498f main
[junit4:junit4]   2> 6485 T86 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 6487 T88 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@20f5498f main{StandardDirectoryReader(segments_7:23:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1 _7(4.1):C1)}
[junit4:junit4]   2> 6487 T86 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6488 T86 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 10
[junit4:junit4]   2> 6491 T99 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6493 T99 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_6,generation=6,filenames=[_3.fld, _1_1.len, _2.si, _5_6.len, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _1.inf, _2_6.len, _3.pst, _1.pst, _2_5.len, _5_1.len, _5_3.len, _0.si, _3_3.len, _1_5.len, _1_16.len, _2_16.len, _2.fld, _0_5.len, _0_6.len, _5.pst, _2_3.len, _5.fld, _2_1.len, _0.pst, _1_15.len, _0.fld, _3_1.len, _5.si, _2.pst, _3.inf, _5.inf, _1_3.len, _0_3.len, _1_6.len, _2_15.len, _1.si, _3.si, _0.inf, segments_6, _5_5.len]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_7,generation=7,filenames=[_1_1.len, _3.fld, _5_6.len, _7.si, _1.inf, _7_1.len, _2_6.len, _5_1.len, _1_5.len, _2.fld, _7.inf, _0_6.len, _0_5.len, _2_3.len, _5.fld, _5.si, _5.inf, _0_3.len, _1_6.len, _2_15.len, _0.inf, _3.si, _2.si, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _7_1.liv, _3.pst, _7_6.len, _1.pst, _2_5.len, _7.fld, _5_3.len, _0.si, _3_3.len, _1_16.len, _2_16.len, _7.pst, _5.pst, _1_15.len, _0.pst, _2_1.len, _0.fld, _3_1.len, _2.pst, _3.inf, _7_3.len, segments_7, _1_3.len, _1.si, _7_5.len, _5_5.len]
[junit4:junit4]   2> 6494 T99 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 7
[junit4:junit4]   2> 6495 T99 C7 oass.SolrIndexSearcher.<init> Opening Searcher@291b5077 main
[junit4:junit4]   2> 6495 T99 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6495 T102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@291b5077 main{StandardDirectoryReader(segments_6:22:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1 _7(4.1):C2/1)}
[junit4:junit4]   2> 6495 T99 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6496 T99 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 5
[junit4:junit4]   2> 6501 T87 C6 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=107 status=0 QTime=1 
[junit4:junit4]   2> 6510 T101 C7 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=107 status=0 QTime=2 
[junit4:junit4]   2> 6521 T86 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6527 T100 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6531 T87 C6 REQ [collection1] webapp=/solr path=/update params={_version_=3000&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 6536 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6540 T86 C6 REQ [collection1] webapp=/solr path=/update params={_version_=3001&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 6547 T99 C7 REQ [collection1] webapp=/solr path=/update params={_version_=3001&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6551 T85 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6556 T87 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6561 T101 C7 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6564 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:16794/solr] nUpdates=100
[junit4:junit4]   2> 6569 T86 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 6570 T100 C7 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:16794/solr
[junit4:junit4]   2> 6571 T100 C7 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:16794/solrn=2 versions=[3004, 3003]
[junit4:junit4]   2> 6574 T85 C6 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=3004,3003&qt=/get&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6579 T100 C7 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 6579 T100 C7 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:16794/solr&version=2&getVersions=100} status=0 QTime=15 
[junit4:junit4]   2> 6582 T87 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6585 T87 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_7,generation=7,filenames=[_1_1.len, _3.fld, _5_6.len, _7.si, _1.inf, _7_1.len, _2_6.len, _5_1.len, _1_5.len, _2.fld, _7.inf, _0_6.len, _0_5.len, _2_3.len, _5.fld, _5.si, _5.inf, _0_3.len, _1_6.len, _2_15.len, _0.inf, _3.si, _2.si, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _3.pst, _7_6.len, _1.pst, _2_5.len, _7.fld, _5_3.len, _0.si, _3_3.len, _1_16.len, _2_16.len, _7.pst, _5.pst, _1_15.len, _0.pst, _2_1.len, _0.fld, _3_1.len, _2.pst, _3.inf, _7_3.len, segments_7, _1_3.len, _1.si, _7_5.len, _5_5.len]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@e49f9fa lockFactory=org.apache.lucene.store.NativeFSLockFactory@382a7640; maxCacheMB=0.505859375 maxMergeSizeMB=0.1474609375)),segFN=segments_8,generation=8,filenames=[_1_1.len, _3.fld, _5_6.len, _8_1.len, _7.si, _1.inf, _7_1.len, _2_6.len, _8.si, _5_1.len, _8_1.liv, _1_5.len, _2.fld, _7.inf, _0_6.len, _0_5.len, _2_3.len, _5.fld, _8_3.len, _5.si, _5.inf, _0_3.len, _1_6.len, _2_15.len, _8.inf, _0.inf, _3.si, _8_5.len, _2.si, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _3.pst, _7_6.len, _1.pst, _2_5.len, _7.fld, _5_3.len, _8_6.len, _0.si, _3_3.len, _1_16.len, _2_16.len, _7.pst, _5.pst, _8.pst, _1_15.len, _0.pst, _2_1.len, _0.fld, _8.fld, _3_1.len, _2.pst, _3.inf, _7_3.len, segments_8, _1_3.len, _1.si, _7_5.len, _5_5.len]
[junit4:junit4]   2> 6586 T87 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 8
[junit4:junit4]   2> 6587 T87 C6 oass.SolrIndexSearcher.<init> Opening Searcher@4ef56959 main
[junit4:junit4]   2> 6587 T87 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 6587 T88 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ef56959 main{StandardDirectoryReader(segments_8:25:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1 _7(4.1):C1 _8(4.1):C4/1)}
[junit4:junit4]   2> 6587 T87 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6588 T87 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 6
[junit4:junit4]   2> 6592 T99 C7 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6596 T99 C7 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_7,generation=7,filenames=[_1_1.len, _3.fld, _5_6.len, _7.si, _1.inf, _7_1.len, _2_6.len, _5_1.len, _1_5.len, _2.fld, _7.inf, _0_6.len, _0_5.len, _2_3.len, _5.fld, _5.si, _5.inf, _0_3.len, _1_6.len, _2_15.len, _0.inf, _3.si, _2.si, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _7_1.liv, _3.pst, _7_6.len, _1.pst, _2_5.len, _7.fld, _5_3.len, _0.si, _3_3.len, _1_16.len, _2_16.len, _7.pst, _5.pst, _1_15.len, _0.pst, _2_1.len, _0.fld, _3_1.len, _2.pst, _3.inf, _7_3.len, segments_7, _1_3.len, _1.si, _7_5.len, _5_5.len]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@4e8c2f76 lockFactory=org.apache.lucene.store.NativeFSLockFactory@63adf08f; maxCacheMB=0.96875 maxMergeSizeMB=0.0185546875)),segFN=segments_8,generation=8,filenames=[_1_1.len, _3.fld, _5_6.len, _8_1.len, _7.si, _1.inf, _7_1.len, _2_6.len, _8.si, _5_1.len, _1_5.len, _2.fld, _7.inf, _0_6.len, _0_5.len, _2_3.len, _5.fld, _8_3.len, _5.si, _5.inf, _0_3.len, _1_6.len, _2_15.len, _8.inf, _0.inf, _3.si, _8_5.len, _2.si, _1.fld, _3_5.len, _0_1.len, _2.inf, _3_6.len, _7_1.liv, _3.pst, _7_6.len, _1.pst, _2_5.len, _7.fld, _5_3.len, _8_6.len, _0.si, _3_3.len, _1_16.len, _2_16.len, _7.pst, _5.pst, _8.pst, _1_15.len, _0.pst, _2_1.len, _0.fld, _8.fld, _3_1.len, _2.pst, _3.inf, _7_3.len, segments_8, _1_3.len, _1.si, _7_5.len, _5_5.len]
[junit4:junit4]   2> 6597 T99 C7 oasc.SolrDeletionPolicy.updateCommits newest commit = 8
[junit4:junit4]   2> 6598 T99 C7 oass.SolrIndexSearcher.<init> Opening Searcher@1181a6d main
[junit4:junit4]   2> 6598 T99 C7 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6599 T102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1181a6d main{StandardDirectoryReader(segments_8:24:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1 _7(4.1):C2/1 _8(4.1):C3)}
[junit4:junit4]   2> 6599 T99 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6600 T99 C7 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 9
[junit4:junit4]   2> 6605 T86 C6 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=110 status=0 QTime=2 
[junit4:junit4]   2> 6613 T101 C7 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=110 status=0 QTime=2 
[junit4:junit4]   2> 6662 T59 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=388368796
[junit4:junit4]   2> 6662 T59 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2c591171
[junit4:junit4]   2> 6663 T59 oasu.DirectUpdateHandler2.close closing DirectU

[...truncated too long message...]

t4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=64, name=qtp1731015846-64 Acceptor0 SelectChannelConnector@0.0.0.0:12818, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=114, name=qtp1177842774-114 Acceptor0 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    9) Thread[id=67, name=qtp1731015846-67 Acceptor3 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   10) Thread[id=49, name=qtp1149059701-49 Acceptor3 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=115, name=qtp1177842774-115 Acceptor1 SelectChannelConnector@0.0.0.0:56173, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=65, name=qtp1731015846-65 Acceptor1 SelectChannelConnector@0.0.0.0:12818, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=47, name=qtp1149059701-47 Acceptor1 SelectChannelConnector@0.0.0.0:34217, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=117, name=qtp1177842774-117 Acceptor3 SelectChannelConnector@0.0.0.0:56173, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=46, name=qtp1149059701-46 Acceptor0 SelectChannelConnector@0.0.0.0:34217, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=99, name=qtp1861758951-99 Acceptor3 SelectChannelConnector@0.0.0.0:54385, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([AE8CEDF37E49E507]:0)
[junit4:junit4] Completed on J0 in 301.40s, 1 test, 2 errors <<< FAILURES!

[...truncated 158 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 812 ignored

Total time: 20 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 825 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/825/

5 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test:     1) Thread[id=349, name=qtp1637770670-349 Acceptor2 SelectChannelConnector@0.0.0.0:25570, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=300, name=qtp1343863107-300 Acceptor3 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=280, name=qtp329021625-280 Acceptor1 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=330, name=qtp1786675819-330 Acceptor1 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=331, name=qtp1786675819-331 Acceptor2 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=281, name=qtp329021625-281 Acceptor2 SelectChannelConnector@0.0.0.0:42163, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=279, name=qtp329021625-279 Acceptor0 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=299, name=qtp1343863107-299 Acceptor2 SelectChannelConnector@0.0.0.0:11729, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=332, name=qtp1786675819-332 Acceptor3 SelectChannelConnector@0.0.0.0:36472, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=282, name=qtp329021625-282 Acceptor3 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=329, name=qtp1786675819-329 Acceptor0 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=347, name=qtp1637770670-347 Acceptor0 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=348, name=qtp1637770670-348 Acceptor1 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=297, name=qtp1343863107-297 Acceptor0 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=350, name=qtp1637770670-350 Acceptor3 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=298, name=qtp1343863107-298 Acceptor1 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZk2Test: 
   1) Thread[id=349, name=qtp1637770670-349 Acceptor2 SelectChannelConnector@0.0.0.0:25570, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=300, name=qtp1343863107-300 Acceptor3 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=280, name=qtp329021625-280 Acceptor1 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=330, name=qtp1786675819-330 Acceptor1 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=331, name=qtp1786675819-331 Acceptor2 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=281, name=qtp329021625-281 Acceptor2 SelectChannelConnector@0.0.0.0:42163, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=279, name=qtp329021625-279 Acceptor0 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=299, name=qtp1343863107-299 Acceptor2 SelectChannelConnector@0.0.0.0:11729, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=332, name=qtp1786675819-332 Acceptor3 SelectChannelConnector@0.0.0.0:36472, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=282, name=qtp329021625-282 Acceptor3 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=329, name=qtp1786675819-329 Acceptor0 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=347, name=qtp1637770670-347 Acceptor0 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=348, name=qtp1637770670-348 Acceptor1 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=297, name=qtp1343863107-297 Acceptor0 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=350, name=qtp1637770670-350 Acceptor3 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=298, name=qtp1343863107-298 Acceptor1 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([CF19FDB0212886E8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZk2Test

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=349, name=qtp1637770670-349 Acceptor2 SelectChannelConnector@0.0.0.0:25570, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=300, name=qtp1343863107-300 Acceptor3 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=280, name=qtp329021625-280 Acceptor1 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=330, name=qtp1786675819-330 Acceptor1 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=281, name=qtp329021625-281 Acceptor2 SelectChannelConnector@0.0.0.0:42163, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=331, name=qtp1786675819-331 Acceptor2 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=279, name=qtp329021625-279 Acceptor0 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=299, name=qtp1343863107-299 Acceptor2 SelectChannelConnector@0.0.0.0:11729, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=282, name=qtp329021625-282 Acceptor3 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=332, name=qtp1786675819-332 Acceptor3 SelectChannelConnector@0.0.0.0:36472, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=329, name=qtp1786675819-329 Acceptor0 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=347, name=qtp1637770670-347 Acceptor0 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=348, name=qtp1637770670-348 Acceptor1 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=297, name=qtp1343863107-297 Acceptor0 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=350, name=qtp1637770670-350 Acceptor3 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=298, name=qtp1343863107-298 Acceptor1 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=349, name=qtp1637770670-349 Acceptor2 SelectChannelConnector@0.0.0.0:25570, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=300, name=qtp1343863107-300 Acceptor3 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=280, name=qtp329021625-280 Acceptor1 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=330, name=qtp1786675819-330 Acceptor1 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=281, name=qtp329021625-281 Acceptor2 SelectChannelConnector@0.0.0.0:42163, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=331, name=qtp1786675819-331 Acceptor2 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=279, name=qtp329021625-279 Acceptor0 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=299, name=qtp1343863107-299 Acceptor2 SelectChannelConnector@0.0.0.0:11729, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=282, name=qtp329021625-282 Acceptor3 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=332, name=qtp1786675819-332 Acceptor3 SelectChannelConnector@0.0.0.0:36472, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=329, name=qtp1786675819-329 Acceptor0 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=347, name=qtp1637770670-347 Acceptor0 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=348, name=qtp1637770670-348 Acceptor1 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=297, name=qtp1343863107-297 Acceptor0 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=350, name=qtp1637770670-350 Acceptor3 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=298, name=qtp1343863107-298 Acceptor1 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([CF19FDB0212886E8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=49, name=qtp1774560059-49 Acceptor0 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=52, name=qtp1774560059-52 Acceptor3 SelectChannelConnector@0.0.0.0:23753, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=51, name=qtp1774560059-51 Acceptor2 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=116, name=qtp1815182793-116 Acceptor2 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=33, name=qtp2145764153-33 Acceptor2 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=32, name=qtp2145764153-32 Acceptor1 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=31, name=qtp2145764153-31 Acceptor0 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=114, name=qtp1815182793-114 Acceptor0 SelectChannelConnector@0.0.0.0:50614, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=115, name=qtp1815182793-115 Acceptor1 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=34, name=qtp2145764153-34 Acceptor3 SelectChannelConnector@0.0.0.0:27052, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=117, name=qtp1815182793-117 Acceptor3 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=50, name=qtp1774560059-50 Acceptor1 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest: 
   1) Thread[id=49, name=qtp1774560059-49 Acceptor0 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=52, name=qtp1774560059-52 Acceptor3 SelectChannelConnector@0.0.0.0:23753, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=51, name=qtp1774560059-51 Acceptor2 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=116, name=qtp1815182793-116 Acceptor2 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=33, name=qtp2145764153-33 Acceptor2 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=32, name=qtp2145764153-32 Acceptor1 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=31, name=qtp2145764153-31 Acceptor0 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=114, name=qtp1815182793-114 Acceptor0 SelectChannelConnector@0.0.0.0:50614, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=115, name=qtp1815182793-115 Acceptor1 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=34, name=qtp2145764153-34 Acceptor3 SelectChannelConnector@0.0.0.0:27052, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=117, name=qtp1815182793-117 Acceptor3 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=50, name=qtp1774560059-50 Acceptor1 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([CF19FDB0212886E8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=49, name=qtp1774560059-49 Acceptor0 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=114, name=qtp1815182793-114 Acceptor0 SelectChannelConnector@0.0.0.0:50614, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=52, name=qtp1774560059-52 Acceptor3 SelectChannelConnector@0.0.0.0:23753, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=51, name=qtp1774560059-51 Acceptor2 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=116, name=qtp1815182793-116 Acceptor2 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=115, name=qtp1815182793-115 Acceptor1 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=34, name=qtp2145764153-34 Acceptor3 SelectChannelConnector@0.0.0.0:27052, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=117, name=qtp1815182793-117 Acceptor3 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=33, name=qtp2145764153-33 Acceptor2 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=32, name=qtp2145764153-32 Acceptor1 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=31, name=qtp2145764153-31 Acceptor0 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=50, name=qtp1774560059-50 Acceptor1 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=49, name=qtp1774560059-49 Acceptor0 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=114, name=qtp1815182793-114 Acceptor0 SelectChannelConnector@0.0.0.0:50614, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=52, name=qtp1774560059-52 Acceptor3 SelectChannelConnector@0.0.0.0:23753, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=51, name=qtp1774560059-51 Acceptor2 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=116, name=qtp1815182793-116 Acceptor2 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=115, name=qtp1815182793-115 Acceptor1 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=34, name=qtp2145764153-34 Acceptor3 SelectChannelConnector@0.0.0.0:27052, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=117, name=qtp1815182793-117 Acceptor3 SelectChannelConnector@0.0.0.0:50614, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=33, name=qtp2145764153-33 Acceptor2 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=32, name=qtp2145764153-32 Acceptor1 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=31, name=qtp2145764153-31 Acceptor0 SelectChannelConnector@0.0.0.0:27052, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=50, name=qtp1774560059-50 Acceptor1 SelectChannelConnector@0.0.0.0:23753, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([CF19FDB0212886E8]:0)


FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
Shard still reported as live in zk

Stack Trace:
java.lang.AssertionError: Shard still reported as live in zk
	at __randomizedtesting.SeedInfo.seed([CF19FDB0212886E8:4EFF73A85677E6D4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitToSeeNotLive(AbstractFullDistribZkTestBase.java:1218)
	at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:190)
	at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:83)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:695)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 7700 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4]   2> 7 T15 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-RecoveryZkTest-1351728409873
[junit4:junit4]   2> 15 T15 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 193 T16 oaz.Environment.logEnv Server environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT
[junit4:junit4]   2> 194 T16 oaz.Environment.logEnv Server environment:host.name=lucene.zones.apache.org
[junit4:junit4]   2> 195 T16 oaz.Environment.logEnv Server environment:java.version=1.6.0_32
[junit4:junit4]   2> 196 T16 oaz.Environment.logEnv Server environment:java.vendor=Sun Microsystems Inc.
[junit4:junit4]   2> 196 T16 oaz.Environment.logEnv Server environment:java.home=/usr/local/openjdk6/jre
[junit4:junit4]   2> 197 T16 oaz.Environment.logEnv Server environment:java.class.path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-solrj/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/common/lucene-analyzers-common-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/highlighter/lucene-highlighter-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/memory/lucene-memory-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/misc/lucene-misc-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/spatial/lucene-spatial-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/suggest/lucene-suggest-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/grouping/lucene-grouping-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queries/lucene-queries-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queryparser/lucene-queryparser-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-codec-1.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-lang-2.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/guava-r05.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/javax.servlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/spatial4j-0.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/commons-io-2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpcore-4.1.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/log4j-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-jdk14-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/zookeeper-3.3.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-continuation-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-deploy-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-http-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-io-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-jmx-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-security-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-server-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-servlet-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-util-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-webapp-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-xml-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/servlet-api-3.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/randomizedtesting-runner-2.0.4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.2.0.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-log4j.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-logging.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-oro.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-javamail.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bcel.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jdepend.jar:/usr/local/openjdk6/lib/tools.jar:/home/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.0.4.jar
[junit4:junit4]   2> 200 T16 oaz.Environment.logEnv Server environment:java.library.path=/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjdk6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/local/lib
[junit4:junit4]   2> 200 T16 oaz.Environment.logEnv Server environment:java.io.tmpdir=.
[junit4:junit4]   2> 201 T16 oaz.Environment.logEnv Server environment:java.compiler=<NA>
[junit4:junit4]   2> 202 T16 oaz.Environment.logEnv Server environment:os.name=FreeBSD
[junit4:junit4]   2> 202 T16 oaz.Environment.logEnv Server environment:os.arch=amd64
[junit4:junit4]   2> 203 T16 oaz.Environment.logEnv Server environment:os.version=9.0-RELEASE
[junit4:junit4]   2> 203 T16 oaz.Environment.logEnv Server environment:user.name=hudson
[junit4:junit4]   2> 204 T16 oaz.Environment.logEnv Server environment:user.home=/home/hudson
[junit4:junit4]   2> 204 T16 oaz.Environment.logEnv Server environment:user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 283 T16 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 342 T16 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 425 T15 oasc.ZkTestServer.run start zk server on port:11196
[junit4:junit4]   2> 463 T16 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 576 T15 oaz.Environment.logEnv Client environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT
[junit4:junit4]   2> 577 T15 oaz.Environment.logEnv Client environment:host.name=lucene.zones.apache.org
[junit4:junit4]   2> 578 T15 oaz.Environment.logEnv Client environment:java.version=1.6.0_32
[junit4:junit4]   2> 578 T15 oaz.Environment.logEnv Client environment:java.vendor=Sun Microsystems Inc.
[junit4:junit4]   2> 579 T15 oaz.Environment.logEnv Client environment:java.home=/usr/local/openjdk6/jre
[junit4:junit4]   2> 580 T15 oaz.Environment.logEnv Client environment:java.class.path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-solrj/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/common/lucene-analyzers-common-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/highlighter/lucene-highlighter-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/memory/lucene-memory-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/misc/lucene-misc-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/spatial/lucene-spatial-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/suggest/lucene-suggest-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/grouping/lucene-grouping-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queries/lucene-queries-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queryparser/lucene-queryparser-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-codec-1.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-lang-2.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/guava-r05.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/javax.servlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/spatial4j-0.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/commons-io-2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpcore-4.1.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/log4j-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-jdk14-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/zookeeper-3.3.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-continuation-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-deploy-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-http-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-io-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-jmx-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-security-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-server-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-servlet-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-util-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-webapp-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-xml-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/servlet-api-3.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/randomizedtesting-runner-2.0.4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.2.0.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-log4j.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-logging.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-oro.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-javamail.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bcel.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jdepend.jar:/usr/local/openjdk6/lib/tools.jar:/home/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.0.4.jar
[junit4:junit4]   2> 593 T15 oaz.Environment.logEnv Client environment:java.library.path=/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjdk6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/local/lib
[junit4:junit4]   2> 594 T15 oaz.Environment.logEnv Client environment:java.io.tmpdir=.
[junit4:junit4]   2> 594 T15 oaz.Environment.logEnv Client environment:java.compiler=<NA>
[junit4:junit4]   2> 595 T15 oaz.Environment.logEnv Client environment:os.name=FreeBSD
[junit4:junit4]   2> 595 T15 oaz.Environment.logEnv Client environment:os.arch=amd64
[junit4:junit4]   2> 596 T15 oaz.Environment.logEnv Client environment:os.version=9.0-RELEASE
[junit4:junit4]   2> 596 T15 oaz.Environment.logEnv Client environment:user.name=hudson
[junit4:junit4]   2> 597 T15 oaz.Environment.logEnv Client environment:user.home=/home/hudson
[junit4:junit4]   2> 597 T15 oaz.Environment.logEnv Client environment:user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 603 T15 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:11196 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@15384ed5
[junit4:junit4]   2> 707 T21 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:11196
[junit4:junit4]   2> 707 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 710 T21 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:11196, initiating session
[junit4:junit4]   2> 711 T17 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:17785
[junit4:junit4]   2> 731 T17 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:17785
[junit4:junit4]   2> 750 T19 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 831 T19 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab94aaf290000 with negotiated timeout 10000 for client /140.211.11.196:17785
[junit4:junit4]   2> 831 T21 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:11196, sessionid = 0x13ab94aaf290000, negotiated timeout = 10000
[junit4:junit4]   2> 839 T22 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15384ed5 name:ZooKeeperConnection Watcher:127.0.0.1:11196 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 839 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 845 T15 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 942 T20 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ab94aaf290000
[junit4:junit4]   2> 943 T22 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 943 T15 oaz.ZooKeeper.close Session: 0x13ab94aaf290000 closed
[junit4:junit4]   2> 944 T17 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:17785 which had sessionid 0x13ab94aaf290000
[junit4:junit4]   2> 945 T15 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:11196/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@31bca1c3
[junit4:junit4]   2> 946 T23 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:11196
[junit4:junit4]   2> 946 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 947 T23 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:11196, initiating session
[junit4:junit4]   2> 948 T17 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:46551
[junit4:junit4]   2> 948 T17 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:46551
[junit4:junit4]   2> 950 T19 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab94aaf290001 with negotiated timeout 10000 for client /140.211.11.196:46551
[junit4:junit4]   2> 951 T23 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:11196, sessionid = 0x13ab94aaf290001, negotiated timeout = 10000
[junit4:junit4]   2> 952 T24 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31bca1c3 name:ZooKeeperConnection Watcher:127.0.0.1:11196/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 952 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 988 T15 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 994 T15 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 1005 T15 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 1010 T15 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 1034 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 1073 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 1172 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 1173 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 1281 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 1282 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 1299 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 1299 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 1304 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 1305 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 1309 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 1310 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 1314 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 1315 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 1319 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 1320 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 1325 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 1325 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 1330 T15 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 1330 T15 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 1335 T20 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ab94aaf290001
[junit4:junit4]   2> 1349 T24 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 1349 T15 oaz.ZooKeeper.close Session: 0x13ab94aaf290001 closed
[junit4:junit4]   2> 1349 T17 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:46551 which had sessionid 0x13ab94aaf290001
[junit4:junit4]   2> 1955 T15 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2271 T15 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:27052
[junit4:junit4]   2> 2340 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2341 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2341 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2419 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2420 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2421 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2421 T15 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2423 T15 oasc.CoreContainer.<init> New CoreContainer 1270144924
[junit4:junit4]   2> 2423 T15 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2424 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2474 T15 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2492 T15 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:11196/solr
[junit4:junit4]   2> 2539 T15 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:11196/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@6dfbe226
[junit4:junit4]   2> 2540 T35 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:11196
[junit4:junit4]   2> 2540 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 2542 T35 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:11196, initiating session
[junit4:junit4]   2> 2542 T17 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:56639
[junit4:junit4]   2> 2543 T17 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:56639
[junit4:junit4]   2> 2570 T19 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab94aaf290002 with negotiated timeout 5000 for client /140.211.11.196:56639
[junit4:junit4]   2> 2570 T35 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:11196, sessionid = 0x13ab94aaf290002, negotiated timeout = 5000
[junit4:junit4]   2> 2571 T36 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6dfbe226 name:ZooKeeperConnection Watcher:127.0.0.1:11196/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 2572 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 2589 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2600 T15 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 2603 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:27052_solr
[junit4:junit4]   2> 2608 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:delete cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:27052_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:27052_solr
[junit4:junit4]   2> 2609 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:27052_solr
[junit4:junit4]   2> 2613 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2613 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2614 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2614 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2615 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2615 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2616 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2616 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2617 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2617 T15 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2634 T15 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 2661 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:delete cxid:0x15 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 2662 T15 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 2665 T15 oasc.Overseer.start Overseer (id=88586873104564226-127.0.0.1:27052_solr-n_0000000000) starting
[junit4:junit4]   2> 2666 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2686 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2688 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:create cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2692 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2693 T42 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 2694 T15 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 2696 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 2734 T41 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 2786 T15 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2787 T15 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 2788 T15 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2788 T15 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 2790 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2792 T15 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2793 T15 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2949 T15 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 3084 T15 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 3185 T15 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 3240 T15 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 4284 T15 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 4301 T15 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 4304 T15 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 4450 T15 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4462 T15 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4467 T15 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4474 T15 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4478 T15 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351728409873/control/data/
[junit4:junit4]   2> 4485 T15 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@39df2964
[junit4:junit4]   2> 4516 T15 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351728409873/control/data/index/
[junit4:junit4]   2> 4517 T15 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351728409873/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 4551 T15 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351728409873/control/data/index forceNew:false
[junit4:junit4]   2> 4624 T15 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351728409873/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@4b61cd25; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4625 T15 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4671 T15 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 4844 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 4845 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 4846 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 4852 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 4936 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 4937 T15 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 4941 T15 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 4942 T15 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 4957 T15 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5002 T15 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 5003 T15 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 5008 T15 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5009 T15 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 5009 T15 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 5010 T15 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 5023 T15 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5023 T15 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5024 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5025 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5026 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5027 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5028 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5029 T15 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5045 T15 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 5061 T15 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 5067 T15 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5071 T15 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 5077 T15 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 5081 T15 oass.SolrIndexSearcher.<init> Opening Searcher@58cbffde main
[junit4:junit4]   2> 5127 T15 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5127 T15 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5129 T15 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 5221 T15 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 5448 T43 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@58cbffde main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5456 T15 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 5457 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:create cxid:0x5f zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 5469 T15 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 5470 T15 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:27052/solr shardId:control_shard
[junit4:junit4]   2> 5480 T15 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 5497 T15 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/control_shard/election
[junit4:junit4]   2> 5517 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:delete cxid:0x6e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 5518 T15 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 5519 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:create cxid:0x6f zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5755 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5756 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:27052_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:27052/solr"}
[junit4:junit4]   2> 5771 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:create cxid:0x76 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 5801 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 6029 T15 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 6030 T15 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 6030 T15 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:27052/solr/collection1/
[junit4:junit4]   2> 6031 T15 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 6032 T15 oasc.SyncStrategy.syncToMe http://127.0.0.1:27052/solr/collection1/ has no replicas
[junit4:junit4]   2> 6033 T15 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:27052/solr/collection1/
[junit4:junit4]   2> 6033 T15 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/control_shard
[junit4:junit4]   2> 6043 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290002 type:create cxid:0x94 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6309 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6332 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 6359 T15 oasc.ZkController.register We are http://127.0.0.1:27052/solr/collection1/ and leader is http://127.0.0.1:27052/solr/collection1/
[junit4:junit4]   2> 6363 T15 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:27052/solr
[junit4:junit4]   2> 6363 T15 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 6364 T15 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6366 T15 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6367 T15 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 6368 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 6420 T15 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 6564 T15 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 6567 T15 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:23753
[junit4:junit4]   2> 6568 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6568 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351728416302
[junit4:junit4]   2> 6568 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351728416302/'
[junit4:junit4]   2> 6603 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 6604 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6604 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351728416302
[junit4:junit4]   2> 6605 T15 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351728416302/solr.xml
[junit4:junit4]   2> 6606 T15 oasc.CoreContainer.<init> New CoreContainer 2050052200
[junit4:junit4]   2> 6607 T15 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351728416302/'
[junit4:junit4]   2> 6607 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351728416302/'
[junit4:junit4]   2> 6638 T15 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 6658 T15 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:11196/solr
[junit4:junit4]   2> 6659 T15 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:11196/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@c7f5bf9
[junit4:junit4]   2> 6660 T53 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:11196
[junit4:junit4]   2> 6660 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6662 T53 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:11196, initiating session
[junit4:junit4]   2> 6662 T17 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:58283
[junit4:junit4]   2> 6663 T17 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:58283
[junit4:junit4]   2> 6666 T19 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab94aaf290003 with negotiated timeout 5000 for client /140.211.11.196:58283
[junit4:junit4]   2> 6666 T53 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:11196, sessionid = 0x13ab94aaf290003, negotiated timeout = 5000
[junit4:junit4]   2> 6667 T54 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c7f5bf9 name:ZooKeeperConnection Watcher:127.0.0.1:11196/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6668 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6669 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290003 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6683 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6685 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:23753_solr
[junit4:junit4]   2> 6686 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290003 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:23753_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:23753_solr
[junit4:junit4]   2> 6692 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:23753_solr
[junit4:junit4]   2> 6696 T36 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 6697 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 6697 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 6698 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 6698 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 6699 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 6699 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 6699 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 6700 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 6700 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 6701 T15 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 6720 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 6727 T15 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351728416302/collection1
[junit4:junit4]   2> 6728 T15 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 6729 T15 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 6729 T15 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 6731 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351728416302/collection1/'
[junit4:junit4]   2> 6733 T15 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty1-1351728416302/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 6733 T15 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty1-1351728416302/collection1/lib/README' to classloader
[junit4:junit4]   2> 6790 T15 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 6839 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6840 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:27052_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:27052/solr"}
[junit4:junit4]   2> 6849 T15 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 6864 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6961 T54 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6961 T15 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 6997 T15 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 7566 T15 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 7580 T15 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 7584 T15 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 7616 T15 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7622 T15 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7627 T15 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7631 T15 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7634 T15 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351728416302/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty1/
[junit4:junit4]   2> 7635 T15 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@39df2964
[junit4:junit4]   2> 7636 T15 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty1/index/
[junit4:junit4]   2> 7637 T15 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 7638 T15 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty1/index forceNew:false
[junit4:junit4]   2> 7671 T15 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@593f5a2f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 7671 T15 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 7672 T15 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 7675 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 7676 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 7677 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 7677 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 7678 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 7679 T15 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 7679 T15 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 7681 T15 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 7682 T15 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 7683 T15 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 7685 T15 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 7685 T15 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 7687 T15 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 7687 T15 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 7688 T15 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 7689 T15 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 7690 T15 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7692 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7693 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7694 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7695 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7697 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7698 T15 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7699 T15 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 7701 T15 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 7707 T15 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 7713 T15 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 7716 T15 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 7721 T15 oass.SolrIndexSearcher.<init> Opening Searcher@4c84f665 main
[junit4:junit4]   2> 7722 T15 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 7723 T15 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7725 T15 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 7733 T15 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 7788 T59 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c84f665 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 7804 T15 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7884 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7886 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:23753_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:23753/solr"}
[junit4:junit4]   2> 7886 T41 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 7907 T54 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 7907 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 8308 T15 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 8309 T15 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:23753/solr shardId:shard1
[junit4:junit4]   2> 8310 T15 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 8335 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290003 type:delete cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 8336 T15 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 8337 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290003 type:create cxid:0x44 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8340 T15 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 8341 T15 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 8342 T15 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:23753/solr/collection1/
[junit4:junit4]   2> 8343 T15 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 8343 T15 oasc.SyncStrategy.syncToMe http://127.0.0.1:23753/solr/collection1/ has no replicas
[junit4:junit4]   2> 8344 T15 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:23753/solr/collection1/
[junit4:junit4]   2> 8345 T15 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 8352 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290003 type:create cxid:0x4e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8413 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8426 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 8427 T54 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 8457 T15 oasc.ZkController.register We are http://127.0.0.1:23753/solr/collection1/ and leader is http://127.0.0.1:23753/solr/collection1/
[junit4:junit4]   2> 8457 T15 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:23753/solr
[junit4:junit4]   2> 8458 T15 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8459 T15 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8461 T15 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8463 T15 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 8464 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8464 T15 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8665 T15 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 8668 T15 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:50614
[junit4:junit4]   2> 8669 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8670 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351728418355
[junit4:junit4]   2> 8671 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351728418355/'
[junit4:junit4]   2> 8711 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8711 T15 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8712 T15 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351728418355
[junit4:junit4]   2> 8713 T15 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351728418355/solr.xml
[junit4:junit4]   2> 8713 T15 oasc.CoreContainer.<init> New CoreContainer 106193257
[junit4:junit4]   2> 8714 T15 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351728418355/'
[junit4:junit4]   2> 8715 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351728418355/'
[junit4:junit4]   2> 8745 T15 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8763 T15 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:11196/solr
[junit4:junit4]   2> 8764 T15 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:11196/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@121ca203
[junit4:junit4]   2> 8765 T69 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:11196
[junit4:junit4]   2> 8765 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8767 T17 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:33520
[junit4:junit4]   2> 8768 T69 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:11196, initiating session
[junit4:junit4]   2> 8769 T17 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:33520
[junit4:junit4]   2> 8779 T19 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab94aaf290004 with negotiated timeout 5000 for client /140.211.11.196:33520
[junit4:junit4]   2> 8779 T69 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:11196, sessionid = 0x13ab94aaf290004, negotiated timeout = 5000
[junit4:junit4]   2> 8780 T70 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@121ca203 name:ZooKeeperConnection Watcher:127.0.0.1:11196/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8780 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8781 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290004 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8797 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290004 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8799 T15 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50614_solr
[junit4:junit4]   2> 8800 T20 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab94aaf290004 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:50614_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:50614_solr
[junit4:junit4]   2> 8801 T15 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50614_solr
[junit4:junit4]   2> 8809 T36 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 8810 T54 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 8811 T15 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 8811 T15 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8812 T15 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 8813 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8813 T15 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8814 T15 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8815 T15 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8815 T15 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8816 T15 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8816 T15 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 8837 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 8846 T15 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351728418355/collection1
[junit4:junit4]   2> 8847 T15 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 8848 T15 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 8849 T15 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 8851 T15 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351728418355/collection1/'
[junit4:junit4]   2> 8852 T15 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty2-1351728418355/collection1/lib/README' to classloader
[junit4:junit4]   2> 8854 T15 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty2-1351728418355/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 8933 T15 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 8936 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8938 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:23753_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:23753/solr"}
[junit4:junit4]   2> 8954 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8954 T70 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8954 T54 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 9023 T15 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 9125 T15 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 9191 T15 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 9982 T15 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 9999 T15 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10005 T15 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10041 T15 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10048 T15 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10055 T15 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10060 T15 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10064 T15 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351728418355/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty2/
[junit4:junit4]   2> 10065 T15 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@39df2964
[junit4:junit4]   2> 10067 T15 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty2/index/
[junit4:junit4]   2> 10068 T15 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10069 T15 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty2/index forceNew:false
[junit4:junit4]   2> 10102 T15 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5fdbe72f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10103 T15 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10104 T15 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 10108 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10108 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 10109 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10111 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 10112 T15 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10112 T15 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10113 T15 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10115 T15 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10117 T15 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10118 T15 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 10120 T15 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 10121 T15 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10122 T15 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 10123 T15 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 10124 T15 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 10126 T15 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10127 T15 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10129 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10131 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10133 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10136 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10139 T15 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10142 T15 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10144 T15 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 10147 T15 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 10159 T15 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10165 T15 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 10167 T15 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 10172 T15 oass.SolrIndexSearcher.<init> Opening Searcher@6cbb3f71 main
[junit4:junit4]   2> 10174 T15 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10174 T15 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10176 T15 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 10184 T15 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 10243 T74 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6cbb3f71 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10251 T15 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 10471 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10473 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50614_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50614/solr"}
[junit4:junit4]   2> 10487 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 10487 T70 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 10487 T54 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 10754 T15 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 10754 T15 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:50614/solr shardId:shard1
[junit4:junit4]   2> 10760 T15 oasc.ZkController.register We are http://127.0.0.1:50614/solr/collection1/ and leader is http://127.0.0.1:23753/solr/collection1/
[junit4:junit4]   2> 10760 T15 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50614/solr
[junit4:junit4]   2> 10761 T15 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 10762 T15 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@7dd41116 url=http://127.0.0.1:50614/solr/collection1 node=127.0.0.1:50614_solr
[junit4:junit4]   2> 10768 T75 C0 P50614 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 10769 T15 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10772 T75 C0 P50614 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 10773 T75 C0 P50614 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 10774 T15 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 10775 T15 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 10776 T15 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 10776 T75 C0 P50614 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:23753/solr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 10787 T75 C0 P50614 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 10797 T15 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:11196/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@42101da9
[junit4:junit4]   2> 10798 T76 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:11196
[junit4:junit4]   2> 10798 T15 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 10800 T76 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:11196, initiating session
[junit4:junit4]   2> 10800 T17 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:53521
[junit4:junit4]   2> 10800 T17 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:53521
[junit4:junit4]   2> 10802 T19 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab94aaf290005 with negotiated timeout 10000 for client /140.211.11.196:53521
[junit4:junit4]   2> 10802 T76 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:11196, sessionid = 0x13ab94aaf290005, negotiated timeout = 10000
[junit4:junit4]   2> 10803 T77 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42101da9 name:ZooKeeperConnection Watcher:127.0.0.1:11196/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 10803 T15 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 10805 T15 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 10805 T75 C0 P50614 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50614/solr START replicas=[http://127.0.0.1:23753/solr/collection1/] nUpdates=100
[junit4:junit4]   2> 10812 T15 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 10823 T15 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 10824 T75 C0 P50614 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 10824 T75 C0 P50614 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 10825 T75 C0 P50614 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 10825 T75 C0 P50614 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 10856 T15 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10993 T41 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10994 T41 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:50614_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:50614/solr"}
[junit4:junit4]   2> 11011 T77 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 11011 T36 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 11011 T70 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 11011 T54 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@11dba9f9 url=http://127.0.0.1:23753/solr/collection1 node=127.0.0.1:23753_solr
[junit4:junit4]   2> 12205 T56 C1 P23753 REQ /get {distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=91 
[junit4:junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@655daa9e url=http://127.0.0.1:27052/solr/collection1 node=127.0.0.1:27052_solr
[junit4:junit4]   2> 12226 T39 C2 P27052 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16
[junit4:junit4]   2> 12682 T38 C2 P27052 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351728409873/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@4b61cd25; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 12683 T38 C2 P27052 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 12964 T39 C2 P27052 /update {wt=javabin&version=2} {add=[10001 (1417389982396448769)]} 0 770
[junit4:junit4]   2> 12964 T38 C2 P27052 /update {wt=javabin&version=2} {add=[1 (1417389982396448768)]} 0 770
[junit4:junit4]   2> 12966 T80 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:11196/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@6b6d079a
[junit4:junit4]   2> 12968 T81 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:11196
[junit4:junit4]   2> 12968 T80 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12968 T81 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:11196, initiating session
[junit4:junit4]   2> 12969 T17 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:63390
[junit4:junit4]   2> 12969 T17 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:63390
[junit4:junit4]   2> 12971 T19 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab94aaf290006 with negotiated timeout 10000 for client /140.211.11.196:63390
[junit4:junit4]   2> 12971 T81 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:11196, sessionid = 0x13ab94aaf290006, negotiated timeout = 10000
[junit4:junit4]   2> 12972 T82 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b6d079a name:ZooKeeperConnection Watcher:127.0.0.1:11196/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12973 T80 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12974 T80 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 13005 T58 C1 P23753 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@593f5a2f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 13006 T58 C1 P23753 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2>  C0_STATE=coll:collection1 core:collection1 props:{shard=shard1, roles=null, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:50614_solr, base_url=http://127.0.0.1:50614/solr}
[junit4:junit4]   2> 13055 T72 C0 P50614 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351728409873/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@5fdbe72f; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 13056 T72 C0 P50614 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 13061 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=10 
[junit4:junit4]   2> 13062 T56 C1 P23753 /update {wt=javabin&version=2} {add=[1 (1417389982736187393)]} 0 65
[junit4:junit4]   2> 13068 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=16 
[junit4:junit4]   2> 13070 T58 C1 P23753 /update {wt=javabin&version=2} {add=[10001 (1417389982736187392)]} 0 74
[junit4:junit4]   2> 13073 T40 C2 P27052 /update {wt=javabin&version=2} {add=[2 (1417389982805393408)]} 0 6
[junit4:junit4]   2> 13076 T39 C2 P27052 /update {wt=javabin&version=2} {add=[10002 (1417389982811684864)]} 0 2
[junit4:junit4]   2> 13087 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 13089 T57 C1 P23753 /update {wt=javabin&version=2} {add=[2 (1417389982814830592)]} 0 12
[junit4:junit4]   2> 13092 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 13093 T56 C1 P23753 /update {wt=javabin&version=2} {add=[10002 (1417389982819024896)]} 0 12
[junit4:junit4]   2> 13097 T38 C2 P27052 /update {wt=javabin&version=2} {add=[3 (1417389982832656384)]} 0 3
[junit4:junit4]   2> 13099 T40 C2 P27052 /update {wt=javabin&version=2} {add=[10003 (1417389982835802112)]} 0 2
[junit4:junit4]   2> 13110 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 13112 T58 C1 P23753 /update {wt=javabin&version=2} {add=[3 (1417389982839996416)]} 0 11
[junit4:junit4]   2> 13114 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 13115 T57 C1 P23753 /update {wt=javabin&version=2} {add=[10003 (1417389982843142144)]} 0 11
[junit4:junit4]   2> 13118 T39 C2 P27052 /update {wt=javabin&version=2} {add=[4 (1417389982855725056)]} 0 2
[junit4:junit4]   2> 13123 T40 C2 P27052 /update {wt=javabin&version=2} {add=[10004 (1417389982859919360)]} 0 3
[junit4:junit4]   2> 13131 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 13132 T58 C1 P23753 /update {wt=javabin&version=2} {add=[4 (1417389982862016512)]} 0 11
[junit4:junit4]   2> 13139 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 13140 T56 C1 P23753 /update {wt=javabin&version=2} {add=[10004 (1417389982867259392)]} 0 13
[junit4:junit4]   2> 13142 T38 C2 P27052 /update {wt=javabin&version=2} {delete=[1 (-1417389982882988032)]} 0 6
[junit4:junit4]   2> 13144 T39 C2 P27052 /update {wt=javabin&version=2} {delete=[10001 (-1417389982885085184)]} 0 0
[junit4:junit4]   2> 13155 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 13155 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 13157 T57 C1 P23753 /update {wt=javabin&version=2} {delete=[1 (-1417389982887182336)]} 0 11
[junit4:junit4]   2> 13157 T58 C1 P23753 /update {wt=javabin&version=2} {delete=[10001 (-1417389982890328064)]} 0 8
[junit4:junit4]   2> 13164 T40 C2 P27052 /update {wt=javabin&version=2} {add=[10005 (1417389982902910976)]} 0 3
[junit4:junit4]   2> 13165 T38 C2 P27052 /update {wt=javabin&version=2} {add=[5 (1417389982902910977)]} 0 4
[junit4:junit4]   2> 13206 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=32 
[junit4:junit4]   2> 13207 T56 C1 P23753 /update {wt=javabin&version=2} {add=[10005 (1417389982910251008)]} 0 39
[junit4:junit4]   2> 13212 T39 C2 P27052 /update {wt=javabin&version=2} {delete=[10003 (-1417389982955339776)]} 0 0
[junit4:junit4]   2> 13220 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 13221 T58 C1 P23753 /update {wt=javabin&version=2} {delete=[10003 (-1417389982960582656)]} 0 5
[junit4:junit4]   2> 13238 T40 C2 P27052 /update {wt=javabin&version=2} {add=[10006 (1417389982971068416)]} 0 12
[junit4:junit4]   2> 13669 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=394 
[junit4:junit4]   2> 13670 T56 C1 P23753 /update {wt=javabin&version=2} {add=[10006 (1417389982988894208)]} 0 427
[junit4:junit4]   2> 13676 T38 C2 P27052 /update {wt=javabin&version=2} {add=[10007 (1417389983440830464)]} 0 3
[junit4:junit4]   2> 13677 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 13678 T57 C1 P23753 /update {wt=javabin&version=2} {add=[5 (1417389982912348160)]} 0 508
[junit4:junit4]   2> 13681 T40 C2 P27052 /update {wt=javabin&version=2} {delete=[3 (-1417389983447121920)]} 0 1
[junit4:junit4]   2> 13687 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 13688 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 13688 T58 C1 P23753 /update {wt=javabin&version=2} {add=[10007 (1417389983447121920)]} 0 9
[junit4:junit4]   2> 13689 T57 C1 P23753 /update {wt=javabin&version=2} {delete=[3 (-1417389983452364800)]} 0 5
[junit4:junit4]   2> 13692 T38 C2 P27052 /update {wt=javabin&version=2} {delete=[10004 (-1417389983458656256)]} 0 1
[junit4:junit4]   2> 13693 T39 C2 P27052 /update {wt=javabin&version=2} {add=[6 (1417389983459704832)]} 0 2
[junit4:junit4]   2> 13698 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 13699 T56 C1 P23753 /update {wt=javabin&version=2} {delete=[10004 (-1417389983462850560)]} 0 5
[junit4:junit4]   2> 13705 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 13706 T58 C1 P23753 /update {wt=javabin&version=2} {add=[6 (1417389983465996288)]} 0 9
[junit4:junit4]   2> 13713 T38 C2 P27052 /update {wt=javabin&version=2} {add=[7 (1417389983478579200)]} 0 4
[junit4:junit4]   2> 13722 T40 C2 P27052 /update {wt=javabin&version=2} {add=[10008 (1417389983471239168)]} 0 20
[junit4:junit4]   2> 13724 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 13725 T57 C1 P23753 /update {wt=javabin&version=2} {add=[7 (1417389983485919232)]} 0 9
[junit4:junit4]   2> 13729 T39 C2 P27052 /update {wt=javabin&version=2} {delete=[4 (-1417389983498502144)]} 0 1
[junit4:junit4]   2> 13733 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 13734 T56 C1 P23753 /update {wt=javabin&version=2} {add=[10008 (1417389983496404992)]} 0 8
[junit4:junit4]   2> 13736 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 13737 T58 C1 P23753 /update {wt=javabin&version=2} {delete=[4 (-1417389983501647872)]} 0 5
[junit4:junit4]   2> 13739 T38 C2 P27052 /update {wt=javabin&version=2} {add=[10009 (1417389983507939328)]} 0 2
[junit4:junit4]   2> 13744 T40 C2 P27052 /update {wt=javabin&version=2} {add=[8 (1417389983511085056)]} 0 4
[junit4:junit4]   2> 13749 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 13750 T57 C1 P23753 /update {wt=javabin&version=2} {add=[10009 (1417389983513182208)]} 0 8
[junit4:junit4]   2> 13756 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 13756 T39 C2 P27052 /update {wt=javabin&version=2} {add=[10010 (1417389983524716544)]} 0 3
[junit4:junit4]   2> 13757 T56 C1 P23753 /update {wt=javabin&version=2} {add=[8 (1417389983518425088)]} 0 10
[junit4:junit4]   2> 13763 T38 C2 P27052 /update {wt=javabin&version=2} {add=[9 (1417389983532056576)]} 0 3
[junit4:junit4]   2> 13767 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 13768 T58 C1 P23753 /update {wt=javabin&version=2} {add=[10010 (1417389983531008000)]} 0 9
[junit4:junit4]   2> 13772 T40 C2 P27052 /update {wt=javabin&version=2} {delete=[10005 (-1417389983543590912)]} 0 0
[junit4:junit4]   2> 13774 T73 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 13775 T57 C1 P23753 /update {wt=javabin&version=2} {add=[9 (1417389983538348032)]} 0 9
[junit4:junit4]   2> 13779 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 13780 T56 C1 P23753 /update {wt=javabin&version=2} {delete=[10005 (-1417389983547785216)]} 0 4
[junit4:junit4]   2> 13781 T39 C2 P27052 /update {wt=javabin&version=2} {add=[10 (1417389983550930944)]} 0 3
[junit4:junit4]   2> 13786 T38 C2 P27052 /update {wt=javabin&version=2} {add=[10011 (1417389983556173824)]} 0 3
[junit4:junit4]   2> 13805 T72 C0 P50614 REQ /update {distrib.from=http://127.0.0.1:23753/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 1

[...truncated too long message...]

nector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=279, name=qtp329021625-279 Acceptor0 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=299, name=qtp1343863107-299 Acceptor2 SelectChannelConnector@0.0.0.0:11729, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    9) Thread[id=282, name=qtp329021625-282 Acceptor3 SelectChannelConnector@0.0.0.0:42163, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   10) Thread[id=332, name=qtp1786675819-332 Acceptor3 SelectChannelConnector@0.0.0.0:36472, state=RUNNABLE, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=329, name=qtp1786675819-329 Acceptor0 SelectChannelConnector@0.0.0.0:36472, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=347, name=qtp1637770670-347 Acceptor0 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=348, name=qtp1637770670-348 Acceptor1 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=297, name=qtp1343863107-297 Acceptor0 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=350, name=qtp1637770670-350 Acceptor3 SelectChannelConnector@0.0.0.0:25570, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=298, name=qtp1343863107-298 Acceptor1 SelectChannelConnector@0.0.0.0:11729, state=BLOCKED, group=TGRP-BasicDistributedZk2Test]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([CF19FDB0212886E8]:0)
[junit4:junit4] Completed on J0 in 308.86s, 1 test, 1 failure, 2 errors <<< FAILURES!

[...truncated 168 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 1 failure, 857 ignored

Total time: 25 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 824 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/824/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedSpellCheckComponentTest

Error Message:
36 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedSpellCheckComponentTest:     1) Thread[id=31, name=qtp1559170267-31 Acceptor2 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=96, name=qtp1421788223-96 Acceptor1 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=18, name=qtp429231097-18 Acceptor2 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=63, name=qtp878602771-63 Acceptor2 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=123, name=qtp542899160-123 Acceptor1 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=20, name=qtp429231097-20 Acceptor3 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=29, name=qtp1559170267-29 Acceptor0 SelectChannelConnector@0.0.0.0:40924, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=125, name=qtp542899160-125 Acceptor3 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=112, name=qtp1506308145-112 Acceptor3 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=75, name=qtp1247765324-75 Acceptor0 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=111, name=qtp1506308145-111 Acceptor2 SelectChannelConnector@0.0.0.0:40957, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=95, name=qtp1421788223-95 Acceptor0 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=17, name=qtp429231097-17 Acceptor1 SelectChannelConnector@0.0.0.0:58853, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=48, name=qtp1155342518-48 Acceptor1 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=49, name=qtp1155342518-49 Acceptor2 SelectChannelConnector@0.0.0.0:40940, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=16, name=qtp429231097-16 Acceptor0 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=50, name=qtp1155342518-50 Acceptor3 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=110, name=qtp1506308145-110 Acceptor1 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=135, name=qtp1620364258-135 Acceptor0 SelectChannelConnector@0.0.0.0:40961, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=47, name=qtp1155342518-47 Acceptor0 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=62, name=qtp878602771-62 Acceptor1 SelectChannelConnector@0.0.0.0:40941, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=138, name=qtp1620364258-138 Acceptor3 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=98, name=qtp1421788223-98 Acceptor3 SelectChannelConnector@0.0.0.0:40955, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=122, name=qtp542899160-122 Acceptor0 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=64, name=qtp878602771-64 Acceptor3 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=97, name=qtp1421788223-97 Acceptor2 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=124, name=qtp542899160-124 Acceptor2 SelectChannelConnector@0.0.0.0:40958, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=32, name=qtp1559170267-32 Acceptor3 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=109, name=qtp1506308145-109 Acceptor0 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=77, name=qtp1247765324-77 Acceptor2 SelectChannelConnector@0.0.0.0:40943, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=78, name=qtp1247765324-78 Acceptor3 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=136, name=qtp1620364258-136 Acceptor1 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=30, name=qtp1559170267-30 Acceptor1 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=61, name=qtp878602771-61 Acceptor0 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=137, name=qtp1620364258-137 Acceptor2 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=76, name=qtp1247765324-76 Acceptor1 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 36 threads leaked from SUITE scope at org.apache.solr.handler.component.DistributedSpellCheckComponentTest: 
   1) Thread[id=31, name=qtp1559170267-31 Acceptor2 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=96, name=qtp1421788223-96 Acceptor1 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=18, name=qtp429231097-18 Acceptor2 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=63, name=qtp878602771-63 Acceptor2 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=123, name=qtp542899160-123 Acceptor1 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=20, name=qtp429231097-20 Acceptor3 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=29, name=qtp1559170267-29 Acceptor0 SelectChannelConnector@0.0.0.0:40924, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=125, name=qtp542899160-125 Acceptor3 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=112, name=qtp1506308145-112 Acceptor3 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=75, name=qtp1247765324-75 Acceptor0 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=111, name=qtp1506308145-111 Acceptor2 SelectChannelConnector@0.0.0.0:40957, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=95, name=qtp1421788223-95 Acceptor0 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=17, name=qtp429231097-17 Acceptor1 SelectChannelConnector@0.0.0.0:58853, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=48, name=qtp1155342518-48 Acceptor1 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=49, name=qtp1155342518-49 Acceptor2 SelectChannelConnector@0.0.0.0:40940, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=16, name=qtp429231097-16 Acceptor0 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=50, name=qtp1155342518-50 Acceptor3 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=110, name=qtp1506308145-110 Acceptor1 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=135, name=qtp1620364258-135 Acceptor0 SelectChannelConnector@0.0.0.0:40961, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=47, name=qtp1155342518-47 Acceptor0 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=62, name=qtp878602771-62 Acceptor1 SelectChannelConnector@0.0.0.0:40941, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=138, name=qtp1620364258-138 Acceptor3 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=98, name=qtp1421788223-98 Acceptor3 SelectChannelConnector@0.0.0.0:40955, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=122, name=qtp542899160-122 Acceptor0 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=64, name=qtp878602771-64 Acceptor3 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=97, name=qtp1421788223-97 Acceptor2 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=124, name=qtp542899160-124 Acceptor2 SelectChannelConnector@0.0.0.0:40958, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=32, name=qtp1559170267-32 Acceptor3 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=109, name=qtp1506308145-109 Acceptor0 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=77, name=qtp1247765324-77 Acceptor2 SelectChannelConnector@0.0.0.0:40943, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=78, name=qtp1247765324-78 Acceptor3 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=136, name=qtp1620364258-136 Acceptor1 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=30, name=qtp1559170267-30 Acceptor1 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=61, name=qtp878602771-61 Acceptor0 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=137, name=qtp1620364258-137 Acceptor2 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=76, name=qtp1247765324-76 Acceptor1 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([4CE5488448EA3595]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.DistributedSpellCheckComponentTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=31, name=qtp1559170267-31 Acceptor2 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=96, name=qtp1421788223-96 Acceptor1 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=18, name=qtp429231097-18 Acceptor2 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=63, name=qtp878602771-63 Acceptor2 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=123, name=qtp542899160-123 Acceptor1 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=20, name=qtp429231097-20 Acceptor3 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=29, name=qtp1559170267-29 Acceptor0 SelectChannelConnector@0.0.0.0:40924, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=125, name=qtp542899160-125 Acceptor3 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=112, name=qtp1506308145-112 Acceptor3 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=75, name=qtp1247765324-75 Acceptor0 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=111, name=qtp1506308145-111 Acceptor2 SelectChannelConnector@0.0.0.0:40957, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=95, name=qtp1421788223-95 Acceptor0 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=17, name=qtp429231097-17 Acceptor1 SelectChannelConnector@0.0.0.0:58853, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=48, name=qtp1155342518-48 Acceptor1 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=49, name=qtp1155342518-49 Acceptor2 SelectChannelConnector@0.0.0.0:40940, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=16, name=qtp429231097-16 Acceptor0 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=50, name=qtp1155342518-50 Acceptor3 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=110, name=qtp1506308145-110 Acceptor1 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=47, name=qtp1155342518-47 Acceptor0 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=135, name=qtp1620364258-135 Acceptor0 SelectChannelConnector@0.0.0.0:40961, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=62, name=qtp878602771-62 Acceptor1 SelectChannelConnector@0.0.0.0:40941, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=98, name=qtp1421788223-98 Acceptor3 SelectChannelConnector@0.0.0.0:40955, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=138, name=qtp1620364258-138 Acceptor3 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=122, name=qtp542899160-122 Acceptor0 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=64, name=qtp878602771-64 Acceptor3 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=97, name=qtp1421788223-97 Acceptor2 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=124, name=qtp542899160-124 Acceptor2 SelectChannelConnector@0.0.0.0:40958, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=32, name=qtp1559170267-32 Acceptor3 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=109, name=qtp1506308145-109 Acceptor0 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=77, name=qtp1247765324-77 Acceptor2 SelectChannelConnector@0.0.0.0:40943, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=78, name=qtp1247765324-78 Acceptor3 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=136, name=qtp1620364258-136 Acceptor1 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=30, name=qtp1559170267-30 Acceptor1 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=61, name=qtp878602771-61 Acceptor0 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=137, name=qtp1620364258-137 Acceptor2 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=76, name=qtp1247765324-76 Acceptor1 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=31, name=qtp1559170267-31 Acceptor2 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=96, name=qtp1421788223-96 Acceptor1 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=18, name=qtp429231097-18 Acceptor2 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=63, name=qtp878602771-63 Acceptor2 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=123, name=qtp542899160-123 Acceptor1 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=20, name=qtp429231097-20 Acceptor3 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=29, name=qtp1559170267-29 Acceptor0 SelectChannelConnector@0.0.0.0:40924, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=125, name=qtp542899160-125 Acceptor3 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=112, name=qtp1506308145-112 Acceptor3 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=75, name=qtp1247765324-75 Acceptor0 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=111, name=qtp1506308145-111 Acceptor2 SelectChannelConnector@0.0.0.0:40957, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=95, name=qtp1421788223-95 Acceptor0 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=17, name=qtp429231097-17 Acceptor1 SelectChannelConnector@0.0.0.0:58853, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=48, name=qtp1155342518-48 Acceptor1 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=49, name=qtp1155342518-49 Acceptor2 SelectChannelConnector@0.0.0.0:40940, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=16, name=qtp429231097-16 Acceptor0 SelectChannelConnector@0.0.0.0:58853, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=50, name=qtp1155342518-50 Acceptor3 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=110, name=qtp1506308145-110 Acceptor1 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=47, name=qtp1155342518-47 Acceptor0 SelectChannelConnector@0.0.0.0:40940, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=135, name=qtp1620364258-135 Acceptor0 SelectChannelConnector@0.0.0.0:40961, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=62, name=qtp878602771-62 Acceptor1 SelectChannelConnector@0.0.0.0:40941, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=98, name=qtp1421788223-98 Acceptor3 SelectChannelConnector@0.0.0.0:40955, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=138, name=qtp1620364258-138 Acceptor3 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=122, name=qtp542899160-122 Acceptor0 SelectChannelConnector@0.0.0.0:40958, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=64, name=qtp878602771-64 Acceptor3 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=97, name=qtp1421788223-97 Acceptor2 SelectChannelConnector@0.0.0.0:40955, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=124, name=qtp542899160-124 Acceptor2 SelectChannelConnector@0.0.0.0:40958, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=32, name=qtp1559170267-32 Acceptor3 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=109, name=qtp1506308145-109 Acceptor0 SelectChannelConnector@0.0.0.0:40957, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=77, name=qtp1247765324-77 Acceptor2 SelectChannelConnector@0.0.0.0:40943, state=RUNNABLE, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=78, name=qtp1247765324-78 Acceptor3 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=136, name=qtp1620364258-136 Acceptor1 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=30, name=qtp1559170267-30 Acceptor1 SelectChannelConnector@0.0.0.0:40924, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=61, name=qtp878602771-61 Acceptor0 SelectChannelConnector@0.0.0.0:40941, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=137, name=qtp1620364258-137 Acceptor2 SelectChannelConnector@0.0.0.0:40961, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=76, name=qtp1247765324-76 Acceptor1 SelectChannelConnector@0.0.0.0:40943, state=BLOCKED, group=TGRP-DistributedSpellCheckComponentTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([4CE5488448EA3595]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.request.TestRemoteStreaming

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming:     1) Thread[id=110, name=qtp751313391-110 Acceptor3 SelectChannelConnector@0.0.0.0:63583, state=RUNNABLE, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=108, name=qtp751313391-108 Acceptor1 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=107, name=qtp751313391-107 Acceptor0 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=109, name=qtp751313391-109 Acceptor2 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming: 
   1) Thread[id=110, name=qtp751313391-110 Acceptor3 SelectChannelConnector@0.0.0.0:63583, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=108, name=qtp751313391-108 Acceptor1 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=107, name=qtp751313391-107 Acceptor0 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=109, name=qtp751313391-109 Acceptor2 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([4CE5488448EA3595]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.request.TestRemoteStreaming

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=110, name=qtp751313391-110 Acceptor3 SelectChannelConnector@0.0.0.0:63583, state=RUNNABLE, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=108, name=qtp751313391-108 Acceptor1 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=107, name=qtp751313391-107 Acceptor0 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=109, name=qtp751313391-109 Acceptor2 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=110, name=qtp751313391-110 Acceptor3 SelectChannelConnector@0.0.0.0:63583, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=108, name=qtp751313391-108 Acceptor1 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=107, name=qtp751313391-107 Acceptor0 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=109, name=qtp751313391-109 Acceptor2 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([4CE5488448EA3595]:0)




Build Log:
[...truncated 7679 lines...]
[junit4:junit4] Suite: org.apache.solr.handler.component.DistributedSpellCheckComponentTest
[junit4:junit4]   2> 209 T10 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 701 T10 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 990 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:58853
[junit4:junit4]   2> 1139 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1140 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1141 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1702 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1706 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1706 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1734 T10 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1738 T10 oasc.CoreContainer.<init> New CoreContainer 386765137
[junit4:junit4]   2> 1739 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1740 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1822 T10 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1904 T10 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1906 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1907 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1908 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2084 T10 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2286 T10 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2314 T10 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2373 T10 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 3962 T10 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 3987 T10 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 3991 T10 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 4297 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4309 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4314 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4330 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4414 T10 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/control/data/
[junit4:junit4]   2> 4418 T10 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f248f2b
[junit4:junit4]   2> 4635 T10 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/control/data/index/
[junit4:junit4]   2> 4636 T10 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 4711 T10 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/control/data/index forceNew:false
[junit4:junit4]   2> 5054 T10 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/control/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5055 T10 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5088 T10 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 5991 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5992 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 5992 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 6000 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 6097 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 6109 T10 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 6113 T10 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 6138 T10 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6156 T10 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 6210 T10 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 6212 T10 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 6217 T10 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 6218 T10 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 6219 T10 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 6220 T10 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 6248 T10 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 6249 T10 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 6250 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 6251 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 6252 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 6253 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 6254 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 6255 T10 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 6273 T10 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 6291 T10 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 6412 T10 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 6433 T10 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 6441 T10 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-REPLICATION2.XML, SCHEMA-MINIMAL.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, CURRENCY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-MERGEPOLICY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, ELEVATE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SOLRCONFIG-ENABLEPLUGIN.XML, HYPHENATION.DTD, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, SCHEMA-REVERSED.XML, STOPTYPES-2.TXT, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-SPELLCHECKER.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, SCHEMA-XINCLUDE.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA_CODEC.XML, SCHEMA.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 6496 T10 oass.SolrIndexSearcher.<init> Opening Searcher@469e0c2d main
[junit4:junit4]   2> 6561 T10 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6562 T10 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 6563 T10 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 6728 T10 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 7088 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 7089 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 7089 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 7090 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 7091 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 7092 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 7092 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 7093 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 7094 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 7143 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 7754 T23 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@469e0c2d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 7763 T10 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 7764 T10 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 7764 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 7846 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 7875 T10 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 7878 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:40924
[junit4:junit4]   2> 7879 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7880 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 7880 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7911 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 7912 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 7912 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 7913 T10 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 7913 T10 oasc.CoreContainer.<init> New CoreContainer 791561201
[junit4:junit4]   2> 7914 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7914 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 7941 T10 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 7965 T10 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 7966 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 7967 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 7968 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 8054 T10 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 8127 T10 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 8128 T10 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 8168 T10 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 8950 T10 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 8964 T10 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 8967 T10 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 8985 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 8989 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 8994 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 8996 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 8999 T10 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/shard0/data/
[junit4:junit4]   2> 8999 T10 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f248f2b
[junit4:junit4]   2> 9001 T10 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/shard0/data/index/
[junit4:junit4]   2> 9002 T10 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 9003 T10 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/shard0/data/index forceNew:false
[junit4:junit4]   2> 9009 T10 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/shard0/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9009 T10 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 9010 T10 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 9013 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 9014 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 9014 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 9015 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 9016 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 9016 T10 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 9017 T10 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 9018 T10 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 9019 T10 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 9021 T10 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 9022 T10 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 9023 T10 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 9024 T10 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 9024 T10 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 9025 T10 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 9026 T10 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 9028 T10 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9029 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9030 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9031 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9033 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9034 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9035 T10 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 9036 T10 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 9037 T10 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 9043 T10 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 9047 T10 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 9049 T10 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-REPLICATION2.XML, SCHEMA-MINIMAL.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, CURRENCY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-MERGEPOLICY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, ELEVATE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SOLRCONFIG-ENABLEPLUGIN.XML, HYPHENATION.DTD, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, SCHEMA-REVERSED.XML, STOPTYPES-2.TXT, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-SPELLCHECKER.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, SCHEMA-XINCLUDE.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA_CODEC.XML, SCHEMA.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, STOP-SNOWBALL.TXT, BAD-SCHEMA-SIM-GLOBAL-VS-FT-MISMATCH.XML, PROTWORDS.TXT, SCHEMA-TRIE.XML, SOLRCONFIG_CODEC.XML, BAD-SOLRCONFIG-INVALID-SCRIPTFILE.XML, JASUGGEST.TXT, SCHEMA-TFIDF.XML, SOLRCONFIG-SCRIPT-UPDATEPROCESSOR.XML, SCHEMA-LMJELINEKMERCER.XML, PHRASESUGGEST.TXT, BAD-SOLRCONFIG-MISSING-SCRIPTFILE.XML, SOLRCONFIG-BASIC-LUCENEVERSION31.XML, BAD-SCHEMA-UNIQUEKEY-USES-DEFAULT.XML, OLD_SYNONYMS.TXT, SOLRCONFIG-DELPOLICY2.XML, SOLRCONFIG-NATIVELOCK.XML, XSLT, BAD-SCHEMA-DUP-FIELD.XML, SOLRCONFIG-NOCACHE.XML, SCHEMA-BM25.XML, ADDFIELDS.UPDATEPROCESSOR.JS, SOLRCONFIG-QUERYSENDER-NOQUERY.XML, SOLRCONFIG-ALTDIRECTORY.XML, COMPOUNDDICTIONARY.TXT, SOLRCONFIG_PERF.XML, SOLRCONFIG-INDEXCONFIG.XML, BAD-SCHEMA-ANALYZER-CLASS-AND-NESTED.XML, SCHEMA-NOT-REQUIRED-UNIQUE-KEY.XML, KEEP-2.TXT, SCHEMA12.XML, BAD-SCHEMA-UNIQUEKEY-IS-COPYFIELD-DEST.XML, MAPPING-ISOLATIN1ACCENT.TXT, BAD_SOLRCONFIG.XML, BAD-SCHEMA-EXTERNAL-FILEFIELD.XML]
[junit4:junit4]   2> 9054 T10 oass.SolrIndexSearcher.<init> Opening Searcher@14a97f68 main
[junit4:junit4]   2> 9055 T10 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 9055 T10 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 9057 T10 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 9065 T10 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 9140 T10 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 9140 T10 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 9141 T10 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 9141 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 9142 T10 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 9142 T10 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 9143 T10 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 9143 T10 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 9143 T10 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 9144 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 9169 T37 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@14a97f68 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 9177 T10 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 9177 T10 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 9178 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 9178 T10 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@7eac4d3b
[junit4:junit4]   2> 10955 T21 C0 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 10962 T21 C0 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/control/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10963 T21 C0 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10967 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 609
[junit4:junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@5945a5a
[junit4:junit4]   2> 11117 T36 C1 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 11121 T36 C1 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/shard0/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 11122 T36 C1 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 11123 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 6
[junit4:junit4]   2> 11340 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 207
[junit4:junit4]   2> 11348 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1]} 0 4
[junit4:junit4]   2> 11355 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 2
[junit4:junit4]   2> 11362 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[2]} 0 3
[junit4:junit4]   2> 11367 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 1
[junit4:junit4]   2> 11372 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[3]} 0 2
[junit4:junit4]   2> 11378 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 1
[junit4:junit4]   2> 11383 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[4]} 0 2
[junit4:junit4]   2> 11388 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 2
[junit4:junit4]   2> 11393 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5]} 0 1
[junit4:junit4]   2> 11409 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 1
[junit4:junit4]   2> 11414 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[6]} 0 2
[junit4:junit4]   2> 11419 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 2
[junit4:junit4]   2> 11425 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7]} 0 2
[junit4:junit4]   2> 11430 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 2
[junit4:junit4]   2> 11435 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[8]} 0 2
[junit4:junit4]   2> 11440 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 2
[junit4:junit4]   2> 11445 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[9]} 0 2
[junit4:junit4]   2> 11590 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 142
[junit4:junit4]   2> 11607 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[10]} 0 13
[junit4:junit4]   2> 11613 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 2
[junit4:junit4]   2> 11619 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[12]} 0 3
[junit4:junit4]   2> 11624 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[13]} 0 2
[junit4:junit4]   2> 11629 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[13]} 0 2
[junit4:junit4]   2> 11634 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 2
[junit4:junit4]   2> 11640 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[14]} 0 2
[junit4:junit4]   2> 11645 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 2
[junit4:junit4]   2> 11650 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[15]} 0 2
[junit4:junit4]   2> 11655 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 2
[junit4:junit4]   2> 11660 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[16]} 0 2
[junit4:junit4]   2> 11665 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 2
[junit4:junit4]   2> 11670 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[17]} 0 2
[junit4:junit4]   2> 11675 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 2
[junit4:junit4]   2> 11681 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[18]} 0 3
[junit4:junit4]   2> 11686 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 2
[junit4:junit4]   2> 11691 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[19]} 0 2
[junit4:junit4]   2> 11696 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 1
[junit4:junit4]   2> 11701 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[20]} 0 2
[junit4:junit4]   2> 11713 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 8
[junit4:junit4]   2> 11727 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[21]} 0 12
[junit4:junit4]   2> 11735 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 2
[junit4:junit4]   2> 11742 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[22]} 0 3
[junit4:junit4]   2> 11748 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 3
[junit4:junit4]   2> 11754 T35 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[23]} 0 3
[junit4:junit4]   2> 11759 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 2
[junit4:junit4]   2> 11764 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[24]} 0 2
[junit4:junit4]   2> 11769 T21 C0 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 1
[junit4:junit4]   2> 11775 T34 C1 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25]} 0 3
[junit4:junit4]   2> 11804 T38 C0 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 12250 T38 C0 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/control/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/control/data/index,segFN=segments_2,generation=2,filenames=[_2.si, _0.tis, _1.frq, _2.tii, _1.fnm, _1.tii, _0.si, _0.prx, _0.nrm, _1.tis, _0.fnm, _2.prx, _2.frq, _2.fdt, _2.fdx, _2.fnm, _1.fdx, _1.prx, _2.tis, _0.tii, _1.fdt, _1.si, _0.frq, segments_2, _0.fdx, _2.nrm, _0.fdt, _1.nrm]
[junit4:junit4]   2> 12251 T38 C0 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 12340 T38 C0 oass.SolrIndexSearcher.<init> Opening Searcher@779a639b main
[junit4:junit4]   2> 12344 T38 C0 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 12357 T23 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@779a639b main{StandardDirectoryReader(segments_2:8 _0(4.1):C10 _1(4.1):C10 _2(4.1):C4)}
[junit4:junit4]   2> 12359 T38 C0 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 558
[junit4:junit4]   2> 12363 T36 C1 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 12938 T36 C1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/shard0/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/shard0/data/index,segFN=segments_2,generation=2,filenames=[_2.si, _0.tis, _1.frq, _2.tii, _1.fnm, _1.tii, _0.si, _0.prx, _0.nrm, _1.tis, _0.fnm, _2.prx, _2.frq, _2.fdt, _2.fdx, _2.fnm, _1.fdx, _1.prx, _2.tis, _0.tii, _1.fdt, _1.si, _0.frq, segments_2, _0.fdx, _2.nrm, _0.fdt, _1.nrm]
[junit4:junit4]   2> 12939 T36 C1 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 12955 T36 C1 oass.SolrIndexSearcher.<init> Opening Searcher@1a383244 main
[junit4:junit4]   2> 12956 T36 C1 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 12963 T37 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a383244 main{StandardDirectoryReader(segments_2:8 _0(4.1):C10 _1(4.1):C10 _2(4.1):C4)}
[junit4:junit4]   2> 12965 T36 C1 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 602
[junit4:junit4]   2> 13308 T38 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&q=*:*&qt=spellCheckCompRH&wt=javabin&spellcheck.build=true&version=2} hits=24 status=0 QTime=299 
[junit4:junit4]   2> 13681 T34 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&spellcheck.extendedResults=false&distrib=false&wt=javabin&rows=10&spellcheck.onlyMorePopular=false&version=2&fl=id,score&shard.url=127.0.0.1:40924/solr&NOW=1351693116768&start=0&q=*:*&spellcheck.count=5&qt=spellCheckCompRH&isShard=true&fsv=true&spellcheck.build=true} hits=24 status=0 QTime=206 
[junit4:junit4]   2> 13734 T34 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.extendedResults=false&shards.qt=spellCheckCompRH&ids=3,2,1,10,7,6,5,4,9,8&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&version=2&shard.url=127.0.0.1:40924/solr&NOW=1351693116768&q=*:*&spellcheck.count=1&qt=spellCheckCompRH&isShard=true&spellcheck.build=true} status=0 QTime=16 
[junit4:junit4]   2> 13744 T35 C1 oashc.SpellCheckComponent.finishStage 127.0.0.1:40924/solr null
[junit4:junit4]   2> 13745 T35 C1 oashc.SpellCheckComponent.finishStage 127.0.0.1:40924/solr null
[junit4:junit4]   2> 13748 T35 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards=127.0.0.1:40924/solr&shards.qt=spellCheckCompRH&q=*:*&qt=spellCheckCompRH&wt=javabin&spellcheck.build=true&version=2} hits=24 status=0 QTime=407 
[junit4:junit4]   2> 13857 T21 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards.qt=spellCheckCompRH&q=*:*&spellcheck.q=toyata&distrib=false&qt=spellCheckCompRH&wt=javabin&version=2} hits=24 status=0 QTime=99 
[junit4:junit4]   2> 14067 T35 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=false&shards.qt=spellCheckCompRH&spellcheck.q=toyata&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=[ff01::114]:33332/solr|127.0.0.1:40924/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr&NOW=1351693117296&start=0&q=*:*&spellcheck.count=5&qt=spellCheckCompRH&isShard=true&fsv=true} hits=24 status=0 QTime=11 
[junit4:junit4]   2> 14077 T36 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.extendedResults=false&shards.qt=spellCheckCompRH&ids=3,2,1,10,7,6,5,4,9,8&spellcheck.q=toyata&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&version=2&fl=id,lowerfilt&shard.url=[ff01::114]:33332/solr|127.0.0.1:40924/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr&NOW=1351693117296&q=*:*&spellcheck.count=1&qt=spellCheckCompRH&isShard=true} status=0 QTime=5 
[junit4:junit4]   2> 14080 T34 C1 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/solr|127.0.0.1:40924/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr {suggestions={toyata={numFound=2,startOffset=0,endOffset=6,suggestion=[toyota, sonata]}}}
[junit4:junit4]   2> 14080 T34 C1 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/solr|127.0.0.1:40924/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr null
[junit4:junit4]   2> 14081 T34 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards=[ff01::114]:33332/solr|127.0.0.1:40924/solr|[ff01::213]:33332/solr|[ff01::083]:33332/solr&shards.qt=spellCheckCompRH&q=*:*&spellcheck.q=toyata&qt=spellCheckCompRH&wt=javabin&version=2} hits=24 status=0 QTime=212 
[junit4:junit4]   2> 14096 T38 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&q=*:*&spellcheck.q=toyata&distrib=false&qt=spellCheckCompRH&wt=javabin&version=2} hits=24 status=0 QTime=12 
[junit4:junit4]   2> 14116 T36 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&spellcheck.q=toyata&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=127.0.0.1:40924/solr&NOW=1351693117527&start=0&q=*:*&spellcheck.count=5&qt=spellCheckCompRH&isShard=true&fsv=true} hits=24 status=0 QTime=12 
[junit4:junit4]   2> 14127 T34 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&ids=3,2,1,10,7,6,5,4,9,8&spellcheck.q=toyata&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&version=2&fl=id,lowerfilt&shard.url=127.0.0.1:40924/solr&NOW=1351693117527&q=*:*&spellcheck.count=1&qt=spellCheckCompRH&isShard=true} status=0 QTime=5 
[junit4:junit4]   2> 14128 T35 C1 oashc.SpellCheckComponent.finishStage 127.0.0.1:40924/solr {suggestions={toyata={numFound=2,startOffset=0,endOffset=6,origFreq=0,suggestion=[{word=toyota,freq=1}, {word=sonata,freq=1}]},correctlySpelled=true}}
[junit4:junit4]   2> 14129 T35 C1 oashc.SpellCheckComponent.finishStage 127.0.0.1:40924/solr null
[junit4:junit4]   2> 14129 T35 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards=127.0.0.1:40924/solr&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&q=*:*&spellcheck.q=toyata&qt=spellCheckCompRH&wt=javabin&version=2} hits=24 status=0 QTime=29 
[junit4:junit4]   2> 14142 T21 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&q=*:*&spellcheck.q=bluo&distrib=false&spellcheck.count=4&qt=spellCheckCompRH&wt=javabin&version=2} hits=24 status=0 QTime=10 
[junit4:junit4]   2> 14159 T35 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&spellcheck.q=bluo&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:40924/solr&NOW=1351693117573&start=0&q=*:*&spellcheck.count=5&qt=spellCheckCompRH&isShard=true&fsv=true} hits=24 status=0 QTime=9 
[junit4:junit4]   2> 14170 T34 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&ids=3,2,1,10,7,6,5,4,9,8&spellcheck.q=bluo&distrib=false&wt=javabin&spellcheck.onlyMorePopular=false&version=2&fl=id,lowerfilt&shard.url=[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:40924/solr&NOW=1351693117573&q=*:*&spellcheck.count=4&qt=spellCheckCompRH&isShard=true} status=0 QTime=5 
[junit4:junit4]   2> 14172 T36 C1 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:40924/solr {suggestions={bluo={numFound=2,startOffset=0,endOffset=4,origFreq=0,suggestion=[{word=blue,freq=1}, {word=glue,freq=1}]},correctlySpelled=true}}
[junit4:junit4]   2> 14172 T36 C1 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:40924/solr null
[junit4:junit4]   2> 14172 T36 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards=[ff01::083]:33332/solr|[ff01::114]:33332/solr|127.0.0.1:40924/solr&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&q=*:*&spellcheck.q=bluo&spellcheck.count=4&qt=spellCheckCompRH&wt=javabin&version=2} hits=24 status=0 QTime=26 
[junit4:junit4]   2> 14295 T21 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&q=The+quick+reb+fox+jumped+over+the+lazy+brown+dogs&distrib=false&spellcheck.count=4&qt=spellCheckCompRH&spellcheck.collate=true&wt=javabin&version=2} hits=0 status=0 QTime=119 
[junit4:junit4]   2> 14400 T35 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&distrib=false&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=[ff01::083]:33332/solr|127.0.0.1:40924/solr|[ff01::114]:33332/solr|[ff01::213]:33332/solr&NOW=1351693117792&start=0&q=The+quick+reb+fox+jumped+over+the+lazy+brown+dogs&spellcheck.count=5&qt=spellCheckCompRH&isShard=true&fsv=true} hits=0 status=0 QTime=30 
[junit4:junit4]   2> 14402 T36 C1 oashc.SpellCheckComponent.finishStage [ff01::083]:33332/solr|127.0.0.1:40924/solr|[ff01::114]:33332/solr|[ff01::213]:33332/solr {suggestions={reb={numFound=1,startOffset=10,endOffset=13,origFreq=0,suggestion=[{word=red,freq=13}]},dogs={numFound=1,startOffset=45,endOffset=49,origFreq=0,suggestion=[{word=dogs.,freq=13}]},correctlySpelled=false,collation=The quick red fox jumped over the lazy brown dogs.}}
[junit4:junit4]   2> 14403 T36 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&fl=id,lowerfilt&shards=[ff01::083]:33332/solr|127.0.0.1:40924/solr|[ff01::114]:33332/solr|[ff01::213]:33332/solr&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&q=The+quick+reb+fox+jumped+over+the+lazy+brown+dogs&spellcheck.count=4&qt=spellCheckCompRH&spellcheck.collate=true&wt=javabin&version=2} hits=0 status=0 QTime=38 
[junit4:junit4]   2> 14445 T38 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.collateExtendedResults=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&spellcheck.maxCollations=10&spellcheck.maxCollationTries=10&distrib=false&wt=javabin&spellcheck.collate=true&version=2&fl=id,lowerfilt&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH} hits=0 status=0 QTime=39 
[junit4:junit4]   2> 14484 T35 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&shards.qt=spellCheckCompRH&spellcheck.maxCollations=10&spellcheck.maxCollationTries=10&distrib=false&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=127.0.0.1:40924/solr|[ff01::083]:33332/solr|[ff01::114]:33332/solr|[ff01::213]:33332/solr&NOW=1351693117880&start=0&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH&isShard=true&fsv=true} hits=0 status=0 QTime=26 
[junit4:junit4]   2> 14486 T34 C1 oashc.SpellCheckComponent.finishStage 127.0.0.1:40924/solr|[ff01::083]:33332/solr|[ff01::114]:33332/solr|[ff01::213]:33332/solr {suggestions={quock={numFound=2,startOffset=12,endOffset=17,origFreq=0,suggestion=[{word=quick,freq=1}, {word=quote,freq=12}]},reb={numFound=1,startOffset=19,endOffset=22,origFreq=0,suggestion=[{word=red,freq=13}]},correctlySpelled=false,collation={collationQuery=lowerfilt:(+quick +red),hits=1,misspellingsAndCorrections={quock=quick,reb=red},collationInternalRank=0},collation={collationQuery=lowerfilt:(+quote +red),hits=12,misspellingsAndCorrections={quock=quote,reb=red},collationInternalRank=1}}}
[junit4:junit4]   2> 14486 T34 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.collateExtendedResults=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&spellcheck.maxCollations=10&spellcheck.maxCollationTries=10&wt=javabin&spellcheck.collate=true&version=2&shards=127.0.0.1:40924/solr|[ff01::083]:33332/solr|[ff01::114]:33332/solr|[ff01::213]:33332/solr&fl=id,lowerfilt&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH} hits=0 status=0 QTime=33 
[junit4:junit4]   2> 14509 T41 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=false&spellcheck.maxCollationTries=10&spellcheck.maxCollations=10&distrib=false&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH} hits=0 status=0 QTime=19 
[junit4:junit4]   2> 14535 T35 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&spellcheck.collateExtendedResults=false&spellcheck.maxCollationTries=10&spellcheck.maxCollations=10&distrib=false&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=127.0.0.1:40924/solr&NOW=1351693117940&start=0&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH&isShard=true&fsv=true} hits=0 status=0 QTime=18 
[junit4:junit4]   2> 14536 T36 C1 oashc.SpellCheckComponent.finishStage 127.0.0.1:40924/solr {suggestions={quock={numFound=2,startOffset=12,endOffset=17,origFreq=0,suggestion=[{word=quick,freq=1}, {word=quote,freq=12}]},reb={numFound=1,startOffset=19,endOffset=22,origFreq=0,suggestion=[{word=red,freq=13}]},correctlySpelled=false,collation=lowerfilt:(+quick +red),collationInternalRank=0,collation=lowerfilt:(+quote +red),collationInternalRank=1}}
[junit4:junit4]   2> 14537 T36 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=false&spellcheck.maxCollationTries=10&spellcheck.maxCollations=10&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&shards=127.0.0.1:40924/solr&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH} hits=0 status=0 QTime=24 
[junit4:junit4]   2> 14556 T21 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=false&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH} hits=0 status=0 QTime=16 
[junit4:junit4]   2> 14581 T36 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&spellcheck.collateExtendedResults=false&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&version=2&fl=id,score&shard.url=[ff01::114]:33332/solr|[ff01::083]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:40924/solr&NOW=1351693117987&start=0&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH&isShard=true&fsv=true} hits=0 status=0 QTime=16 
[junit4:junit4]   2> 14606 T34 C1 oashc.SpellCheckComponent.finishStage [ff01::114]:33332/solr|[ff01::083]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:40924/solr {suggestions={quock={numFound=2,startOffset=12,endOffset=17,origFreq=0,suggestion=[{word=quick,freq=1}, {word=quote,freq=12}]},reb={numFound=1,startOffset=19,endOffset=22,origFreq=0,suggestion=[{word=red,freq=13}]},correctlySpelled=false,collation=lowerfilt:(+quick +red)}}
[junit4:junit4]   2> 14608 T34 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=false&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&shards=[ff01::114]:33332/solr|[ff01::083]:33332/solr|[ff01::213]:33332/solr|127.0.0.1:40924/solr&q=lowerfilt:(%2Bquock+%2Breb)&spellcheck.count=10&qt=spellCheckCompRH} hits=0 status=0 QTime=47 
[junit4:junit4]   2> 14665 T41 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.maxResultsForSuggest=10&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&q=lowerfilt:("quote+red+fox")&spellcheck.count=10&qt=spellCheckCompRH&spellcheck.alternativeTermCount=5} hits=12 status=0 QTime=49 
[junit4:junit4]   2> 14679 T36 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&spellcheck.maxCollationTries=10&distrib=false&version=2&fl=id,score&shard.url=127.0.0.1:40924/solr&NOW=1351693118096&spellcheck.count=10&qt=spellCheckCompRH&fsv=true&spellcheck.alternativeTermCount=5&spellcheck.maxResultsForSuggest=10&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&start=0&q=lowerfilt:("quote+red+fox")&isShard=true} hits=12 status=0 QTime=5 
[junit4:junit4]   2> 14689 T34 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.maxResultsForSuggest=10&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&ids=21,20,19,17,22,18,15,16,13,14&distrib=false&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&version=2&fl=id,lowerfilt&shard.url=127.0.0.1:40924/solr&NOW=1351693118096&q=lowerfilt:("quote+red+fox")&spellcheck.count=10&qt=spellCheckCompRH&isShard=true&spellcheck.alternativeTermCount=5} status=0 QTime=5 
[junit4:junit4]   2> 14692 T35 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.maxResultsForSuggest=10&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&shards=127.0.0.1:40924/solr&q=lowerfilt:("quote+red+fox")&spellcheck.count=10&qt=spellCheckCompRH&spellcheck.alternativeTermCount=5} hits=12 status=0 QTime=23 
[junit4:junit4]   2> 14726 T21 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.maxResultsForSuggest=10&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&q=lowerfilt:("rod+fix")&spellcheck.count=10&qt=spellCheckCompRH&spellcheck.alternativeTermCount=5} hits=1 status=0 QTime=30 
[junit4:junit4]   2> 14756 T34 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckCompRH&spellcheck.maxCollationTries=10&distrib=false&version=2&fl=id,score&shard.url=[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:40924/solr|[ff01::114]:33332/solr&NOW=1351693118157&spellcheck.count=10&qt=spellCheckCompRH&fsv=true&spellcheck.alternativeTermCount=5&spellcheck.maxResultsForSuggest=10&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&rows=10&start=0&q=lowerfilt:("rod+fix")&isShard=true} hits=1 status=0 QTime=22 
[junit4:junit4]   2> 14762 T36 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=false&spellcheck.maxResultsForSuggest=10&spellcheck.extendedResults=true&shards.qt=spellCheckCompRH&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&ids=25&distrib=false&spellcheck.collate=true&wt=javabin&spellcheck.onlyMorePopular=false&version=2&fl=id,lowerfilt&shard.url=[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:40924/solr|[ff01::114]:33332/solr&NOW=1351693118157&q=lowerfilt:("rod+fix")&spellcheck.count=10&qt=spellCheckCompRH&isShard=true&spellcheck.alternativeTermCount=5} status=0 QTime=1 
[junit4:junit4]   2> 14763 T35 C1 oashc.SpellCheckComponent.finishStage [ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:40924/solr|[ff01::114]:33332/solr {suggestions={rod={numFound=2,startOffset=12,endOffset=15,origFreq=1,suggestion=[{word=red,freq=13}, {word=ford,freq=1}]},fix={numFound=1,startOffset=16,endOffset=19,origFreq=1,suggestion=[{word=fox,freq=13}]},correctlySpelled=false,collation={collationQuery=lowerfilt:("red fox"),hits=13,misspellingsAndCorrections={rod=red,fix=fox},collationInternalRank=2}}}
[junit4:junit4]   2> 14763 T35 C1 oashc.SpellCheckComponent.finishStage [ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:40924/solr|[ff01::114]:33332/solr null
[junit4:junit4]   2> 14763 T35 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&spellcheck.maxResultsForSuggest=10&shards.qt=spellCheckCompRH&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=10&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&shards=[ff01::213]:33332/solr|[ff01::083]:33332/solr|127.0.0.1:40924/solr|[ff01::114]:33332/solr&q=lowerfilt:("rod+fix")&spellcheck.count=10&qt=spellCheckCompRH&spellcheck.alternativeTermCount=5} hits=1 status=0 QTime=33 
[junit4:junit4]   2> 14837 T38 C0 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckWithWordbreak&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&q=lowerfilt:(%2Bquock+%2Bredfox+%2Bjum+%2Bped)&spellcheck.count=10&qt=spellCheckWithWordbreak} hits=0 status=0 QTime=70 
[junit4:junit4]   2> 14893 T36 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckWithWordbreak&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&distrib=false&spellcheck.collate=true&wt=javabin&rows=10&version=2&fl=id,score&shard.url=127.0.0.1:40924/solr&NOW=1351693118267&start=0&q=lowerfilt:(%2Bquock+%2Bredfox+%2Bjum+%2Bped)&spellcheck.dictionary=default&spellcheck.dictionary=wordbreak&spellcheck.count=10&qt=spellCheckWithWordbreak&isShard=true&fsv=true} hits=0 status=0 QTime=48 
[junit4:junit4]   2> 14897 T34 C1 oashc.SpellCheckComponent.finishStage 127.0.0.1:40924/solr {suggestions={quock={numFound=2,startOffset=12,endOffset=17,origFreq=0,suggestion=[{word=quick,freq=1}, {word=quote,freq=12}]},redfox={numFound=1,startOffset=19,endOffset=25,origFreq=0,suggestion=[{word=red fox,freq=13}]},ped={numFound=1,startOffset=32,endOffset=35,origFreq=0,suggestion=[{word=red,freq=13}]},jum ped={numFound=1,startOffset=27,endOffset=35,origFreq=0,suggestion=[{word=jumped,freq=13}]},correctlySpelled=false,collation={collationQuery=lowerfilt:(+quick (+red +fox) +jum +red),hits=0,misspellingsAndCorrections={quock=quick,redfox=red fox,ped=red}}}}
[junit4:junit4]   2> 14898 T34 C1 REQ [collection1] webapp=/solr path=/select params={spellcheck=true&shards.qt=spellCheckWithWordbreak&spellcheck.extendedResults=true&spellcheck.collateExtendedResults=true&spellcheck.maxCollationTries=0&spellcheck.maxCollations=1&spellcheck.collate=true&wt=javabin&version=2&fl=id,lowerfilt&shards=127.0.0.1:40924/solr&q=lowerfilt:(%2Bquock+%2Bredfox+%2Bjum+%2Bped)&spellcheck.count=10&qt=spellCheckWithWordbreak} hits=0 status=0 QTime=58 
[junit4:junit4]   2> 14954 T10 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=386765137
[junit4:junit4]   2> 14954 T10 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7eac4d3b
[junit4:junit4]   2> 14966 T10 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=24,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 14967 T10 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 14968 T10 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 14968 T10 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 14970 T10 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 14975 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 15076 T10 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 15100 T10 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=791561201
[junit4:junit4]   2> 15101 T10 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5945a5a
[junit4:junit4]   2> 15121 T10 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=24,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 15121 T10 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 15122 T10 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 15122 T10 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 15124 T10 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 15126 T10 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 15226 T10 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 15228 T10 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 15231 T10 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:40940
[junit4:junit4]   2> 15232 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 15232 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 15233 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 15263 T10 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 15263 T10 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 15264 T10 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 15264 T10 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 15265 T10 oasc.CoreContainer.<init> New CoreContainer 104665766
[junit4:junit4]   2> 15265 T10 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 15266 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 15290 T10 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 15310 T10 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 15311 T10 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 15312 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 15313 T10 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 15372 T10 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 15435 T10 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 15436 T10 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 15468 T10 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 15961 T10 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 15973 T10 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 15976 T10 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 15995 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 16002 T10 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 16008 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 16012 T10 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 16015 T10 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/control/data/
[junit4:junit4]   2> 16016 T10 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f248f2b
[junit4:junit4]   2> 16017 T10 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.component.DistributedSpellCheckComponentTest-1351693103641/control/data/index/
[junit4:junit4]   2> 16019 T10 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 16022 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 16023 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 16024 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 16025 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 16027 T10 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 16027 T10 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 16028 T10 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 16030 T10 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 16031 T10 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 16032 T10 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 16033 T10 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 16034 T10 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 16035 T10 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 16036 T10 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 16037 T10 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 16038 T10 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 16039 T10 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 16040 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 16041 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 16042 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 16043 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 16044 T10 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 16045 T10 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 16046 T10 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 16047 T10 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 16055 T10 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 16058 T10 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 16061 T10 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [THROW.ERROR.ON.ADD.UPDATEPROCESSOR.JS, SOLRCONFIG-HIGHLIGHT.XML, SCHEMA-REQUIRED-FIELDS.XML, SCHEMA-REPLICATION2.XML, SCHEMA-MINIMAL.XML, BAD-SCHEMA-DUP-DYNAMICFIELD.XML, TRIVIAL.UPDATEPROCESSOR1.JS, SOLRCONFIG-CACHING.XML, SOLRCONFIG-REPEATER.XML, CURRENCY.XML, BAD-SCHEMA-NONTEXT-ANALYZER.XML, SOLRCONFIG-MERGEPOLICY.XML, SOLRCONFIG-TLOG.XML, SOLRCONFIG-MASTER.XML, BAD-SCHEMA-UNIQUEKEY-MULTIVALUED.XML, SCHEMA11.XML, SOLRCONFIG-BASIC.XML, DA_COMPOUNDDICTIONARY.TXT, MISSLEADING.EXTENSION.UPDATEPROCESSOR.JS.TXT, SCHEMA-COPYFIELD-TEST.XML, SOLRCONFIG-SLAVE.XML, ELEVATE.XML, SOLRCONFIG-PROPINJECT-INDEXDEFAULT.XML, TRIVIAL.UPDATEPROCESSOR0.JS, BAD-SOLRCONFIG-BOGUS-SCRIPTENGINE-NAME.XML, SCHEMA-CHARFILTERS.XML, SCHEMA-IB.XML, SOLRCONFIG-QUERYSENDER.XML, DA_UTF8.XML, SOLRCONFIG-SNIPPET-PROCESSOR.XML, SCHEMA-REPLICATION1.XML, CONDITIONAL.UPDATEPROCESSOR.JS, MISSING.FUNCTIONS.UPDATEPROCESSOR.JS, SOLRCONFIG-ENABLEPLUGIN.XML, HYPHENATION.DTD, SCHEMA-PHRASESUGGEST.XML, STEMDICT.TXT, HUNSPELL-TEST.AFF, SCHEMA-SNIPPET-TYPE.XML, STOPTYPES-1.TXT, STOPWORDSWRONGENCODING.TXT, SCHEMA-NUMERIC.XML, SOLRCONFIG-TRANSFORMERS.XML, SOLRCONFIG-PROPINJECT.XML, BAD-SCHEMA-NOT-INDEXED-BUT-TF.XML, SOLRCONFIG-SIMPLELOCK.XML, WDFTYPES.TXT, SCHEMA-REVERSED.XML, STOPTYPES-2.TXT, BAD-SCHEMA-CURRENCY-FT-MULTIVALUED.XML, SOLRCONFIG-SPELLCHECKCOMPONENT.XML, SCHEMA-DFR.XML, SOLRCONFIG-PHRASESUGGEST.XML, BAD-SCHEMA-CURRENCY-MULTIVALUED.XML, BAD-SCHEMA-NOT-INDEXED-BUT-POS.XML, KEEP-1.TXT, OPEN-EXCHANGE-RATES.JSON, SCHEMA-SPATIAL.XML, STOPWITHBOM.TXT, SCHEMA-BINARYFIELD.XML, SOLRCONFIG-SPELLCHECKER.XML, SOLRCONFIG-UPDATE-PROCESSOR-CHAINS.XML, BAD-SCHEMA-OMIT-TF-BUT-NOT-POS.XML, SCHEMA-XINCLUDE.XML, BAD-SCHEMA-DUP-FIELDTYPE.XML, SOLRCONFIG-MASTER1.XML, SYNONYMS.TXT, BAD-SCHEMA-CURRENCY-DYNAMIC-MULTIVALUED.XML, SCHEMA_CODEC.XML, SCHEMA.XML, SOLRCONFIG-SOLR-749.XML, SOLRCONFIG-MASTER1-KEEPONEBACKUP.XML, STOP-2.TXT, SOLRCONFIG-FUNCTIONQUERY.XML, SCHEMA-LMDIRICHLET.XML, SOLRCONFIG-TERMINDEX.XML, SOLRCONFIG-ELEVATE.XML, STOPWORDS.TXT, SCHEMA-FOLDING.XML, SCHEMA-STOP-KEEP.XML, BAD-SCHEMA-NOT-INDEXED-BUT-NORMS.XML, SOLRCONFIG-SOLCOREPROPERTIES.XML, STOP-1.TXT, SOLRCONFIG-MASTER2.XML, SCHEMA-SPELLCHECKER.XML, SOLRCONFIG-RESPONSE-LOG-COMPONENT.XML, SOLRCONFIG-LAZYWRITER.XML, SCHEMA-LUCENEMATCHVERSION.XML, FRENCHARTICLES.TXT, BAD-MP-SOLRCONFIG.XML, SCHEMA15.XML, SOLRCONFIG-REQHANDLER.INCL, SCHEMASURROUND.XML, SCHEMA-COLLATEFILTER.XML, SOLRCONFIG-MASTER3.XML, HUNSPELL-TEST.DIC, BAD-SCHEMA-CODEC-GLOBAL-VS-FT-MISMATCH.XML, SOLRCONFIG-XINCLUDE.XML, SOLRCONFIG-DELPOLICY1.XML, SOLRCONFIG-SLAVE1.XML, SCHEMA-SIM.XML, SCHEMA-SNIPPET-FIELD.XML, SCHEMA-COLLATE.XML, ST

[...truncated too long message...]

up=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=ar_BH, timezone=Australia/ACT
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=121180512,total=198574080
[junit4:junit4]   2> NOTE: All tests run in this JVM: [ReturnFieldsTest, TestReversedWildcardFilterFactory, SolrInfoMBeanTest, FileUtilsTest, TestQuerySenderListener, TestFoldingMultitermQuery, TestHashPartitioner, TestValueSourceCache, TestOmitPositions, TestDFRSimilarityFactory, UUIDFieldTest, ZkSolrClientTest, DebugComponentTest, StandardRequestHandlerTest, TermVectorComponentTest, TimeZoneUtilsTest, TestQueryUtils, MultiTermTest, TestRemoteStreaming]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRemoteStreaming -Dtests.seed=4CE5488448EA3595 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_BH -Dtests.timezone=Australia/ACT -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR   0.00s J1 | TestRemoteStreaming (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming: 
[junit4:junit4]    >    1) Thread[id=110, name=qtp751313391-110 Acceptor3 SelectChannelConnector@0.0.0.0:63583, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=108, name=qtp751313391-108 Acceptor1 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=107, name=qtp751313391-107 Acceptor0 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=109, name=qtp751313391-109 Acceptor2 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([4CE5488448EA3595]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=110, name=qtp751313391-110 Acceptor3 SelectChannelConnector@0.0.0.0:63583, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=108, name=qtp751313391-108 Acceptor1 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=107, name=qtp751313391-107 Acceptor0 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=109, name=qtp751313391-109 Acceptor2 SelectChannelConnector@0.0.0.0:63583, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([4CE5488448EA3595]:0)
[junit4:junit4] Completed on J1 in 26.37s, 4 tests, 2 errors <<< FAILURES!

[...truncated 194 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1163: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 811 ignored

Total time: 19 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 823 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/823/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.AnalysisAfterCoreReloadTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.AnalysisAfterCoreReloadTest:     1) Thread[id=67, name=qtp661397909-67 Acceptor1 SelectChannelConnector@0.0.0.0:39176, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=68, name=qtp661397909-68 Acceptor2 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=66, name=qtp661397909-66 Acceptor0 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=69, name=qtp661397909-69 Acceptor3 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.AnalysisAfterCoreReloadTest: 
   1) Thread[id=67, name=qtp661397909-67 Acceptor1 SelectChannelConnector@0.0.0.0:39176, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=68, name=qtp661397909-68 Acceptor2 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=66, name=qtp661397909-66 Acceptor0 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=69, name=qtp661397909-69 Acceptor3 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([D4502F33E1BD685A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.AnalysisAfterCoreReloadTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=67, name=qtp661397909-67 Acceptor1 SelectChannelConnector@0.0.0.0:39176, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=68, name=qtp661397909-68 Acceptor2 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=66, name=qtp661397909-66 Acceptor0 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=69, name=qtp661397909-69 Acceptor3 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=67, name=qtp661397909-67 Acceptor1 SelectChannelConnector@0.0.0.0:39176, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=68, name=qtp661397909-68 Acceptor2 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=66, name=qtp661397909-66 Acceptor0 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=69, name=qtp661397909-69 Acceptor3 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([D4502F33E1BD685A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.PeerSyncTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.update.PeerSyncTest:     1) Thread[id=77, name=qtp58654006-77 Acceptor3 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=46, name=qtp1755031090-46 Acceptor0 SelectChannelConnector@0.0.0.0:29678, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=35, name=qtp525030058-35 Acceptor3 SelectChannelConnector@0.0.0.0:34666, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=61, name=qtp1180675534-61 Acceptor1 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=34, name=qtp525030058-34 Acceptor2 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=32, name=qtp525030058-32 Acceptor0 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=74, name=qtp58654006-74 Acceptor0 SelectChannelConnector@0.0.0.0:16512, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=48, name=qtp1755031090-48 Acceptor2 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=75, name=qtp58654006-75 Acceptor1 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=76, name=qtp58654006-76 Acceptor2 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=60, name=qtp1180675534-60 Acceptor0 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=49, name=qtp1755031090-49 Acceptor3 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=47, name=qtp1755031090-47 Acceptor1 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=62, name=qtp1180675534-62 Acceptor2 SelectChannelConnector@0.0.0.0:32584, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=33, name=qtp525030058-33 Acceptor1 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=63, name=qtp1180675534-63 Acceptor3 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.update.PeerSyncTest: 
   1) Thread[id=77, name=qtp58654006-77 Acceptor3 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=46, name=qtp1755031090-46 Acceptor0 SelectChannelConnector@0.0.0.0:29678, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=35, name=qtp525030058-35 Acceptor3 SelectChannelConnector@0.0.0.0:34666, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=61, name=qtp1180675534-61 Acceptor1 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=34, name=qtp525030058-34 Acceptor2 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=32, name=qtp525030058-32 Acceptor0 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=74, name=qtp58654006-74 Acceptor0 SelectChannelConnector@0.0.0.0:16512, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=48, name=qtp1755031090-48 Acceptor2 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=75, name=qtp58654006-75 Acceptor1 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=76, name=qtp58654006-76 Acceptor2 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=60, name=qtp1180675534-60 Acceptor0 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=49, name=qtp1755031090-49 Acceptor3 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=47, name=qtp1755031090-47 Acceptor1 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=62, name=qtp1180675534-62 Acceptor2 SelectChannelConnector@0.0.0.0:32584, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=33, name=qtp525030058-33 Acceptor1 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=63, name=qtp1180675534-63 Acceptor3 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([D4502F33E1BD685A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.PeerSyncTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=77, name=qtp58654006-77 Acceptor3 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=46, name=qtp1755031090-46 Acceptor0 SelectChannelConnector@0.0.0.0:29678, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=35, name=qtp525030058-35 Acceptor3 SelectChannelConnector@0.0.0.0:34666, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=61, name=qtp1180675534-61 Acceptor1 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=34, name=qtp525030058-34 Acceptor2 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=32, name=qtp525030058-32 Acceptor0 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=48, name=qtp1755031090-48 Acceptor2 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=74, name=qtp58654006-74 Acceptor0 SelectChannelConnector@0.0.0.0:16512, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=75, name=qtp58654006-75 Acceptor1 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=60, name=qtp1180675534-60 Acceptor0 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=76, name=qtp58654006-76 Acceptor2 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=49, name=qtp1755031090-49 Acceptor3 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=47, name=qtp1755031090-47 Acceptor1 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=62, name=qtp1180675534-62 Acceptor2 SelectChannelConnector@0.0.0.0:32584, state=RUNNABLE, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=33, name=qtp525030058-33 Acceptor1 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=63, name=qtp1180675534-63 Acceptor3 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=77, name=qtp58654006-77 Acceptor3 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=46, name=qtp1755031090-46 Acceptor0 SelectChannelConnector@0.0.0.0:29678, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=35, name=qtp525030058-35 Acceptor3 SelectChannelConnector@0.0.0.0:34666, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=61, name=qtp1180675534-61 Acceptor1 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=34, name=qtp525030058-34 Acceptor2 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=32, name=qtp525030058-32 Acceptor0 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=48, name=qtp1755031090-48 Acceptor2 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=74, name=qtp58654006-74 Acceptor0 SelectChannelConnector@0.0.0.0:16512, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=75, name=qtp58654006-75 Acceptor1 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=60, name=qtp1180675534-60 Acceptor0 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=76, name=qtp58654006-76 Acceptor2 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=49, name=qtp1755031090-49 Acceptor3 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=47, name=qtp1755031090-47 Acceptor1 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=62, name=qtp1180675534-62 Acceptor2 SelectChannelConnector@0.0.0.0:32584, state=RUNNABLE, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=33, name=qtp525030058-33 Acceptor1 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=63, name=qtp1180675534-63 Acceptor3 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([D4502F33E1BD685A]:0)




Build Log:
[...truncated 7697 lines...]
[junit4:junit4] Suite: org.apache.solr.update.PeerSyncTest
[junit4:junit4]   2> 5 T25 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 533 T25 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 815 T25 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:34666
[junit4:junit4]   2> 845 T25 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 845 T25 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 846 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 878 T25 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 879 T25 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 879 T25 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 879 T25 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 880 T25 oasc.CoreContainer.<init> New CoreContainer 1064301273
[junit4:junit4]   2> 881 T25 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 881 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 915 T25 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 957 T25 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 958 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 959 T25 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 960 T25 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 999 T25 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1042 T25 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 1043 T25 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1062 T25 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1554 T25 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1566 T25 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1569 T25 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1584 T25 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1588 T25 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1592 T25 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1594 T25 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1596 T25 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351665123359/control/data/
[junit4:junit4]   2> 1596 T25 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 1597 T25 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351665123359/control/data/index/
[junit4:junit4]   2> 1598 T25 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351665123359/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1599 T25 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351665123359/control/data/index forceNew:false
[junit4:junit4]   2> 1602 T25 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f4b2263 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5c4aca27),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1602 T25 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1603 T25 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1604 T25 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1604 T25 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1605 T25 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1610 T25 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1615 T25 oass.SolrIndexSearcher.<init> Opening Searcher@5c9aa764 main
[junit4:junit4]   2> 1615 T25 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1637 T25 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1638 T25 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1639 T25 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1639 T25 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1639 T25 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1640 T25 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1640 T25 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1641 T25 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1641 T25 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1641 T25 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1642 T25 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1642 T25 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1654 T40 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5c9aa764 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1656 T25 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1657 T25 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 1657 T25 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1716 T25 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1736 T25 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1739 T25 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:29678
[junit4:junit4]   2> 1739 T25 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1740 T25 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1740 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1774 T25 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1775 T25 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1775 T25 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1775 T25 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1776 T25 oasc.CoreContainer.<init> New CoreContainer 294576878
[junit4:junit4]   2> 1777 T25 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1777 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1806 T25 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1827 T25 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1828 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1829 T25 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1830 T25 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1868 T25 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1914 T25 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 1915 T25 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1931 T25 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2546 T25 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2564 T25 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2568 T25 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2590 T25 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2597 T25 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2602 T25 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2605 T25 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2608 T25 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351665123359/shard0/data/
[junit4:junit4]   2> 2608 T25 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 2610 T25 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351665123359/shard0/data/index/
[junit4:junit4]   2> 2611 T25 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351665123359/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2612 T25 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351665123359/shard0/data/index forceNew:false
[junit4:junit4]   2> 2616 T25 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2617 T25 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2619 T25 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2620 T25 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2621 T25 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2622 T25 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2632 T25 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2639 T25 oass.SolrIndexSearcher.<init> Opening Searcher@29578426 main
[junit4:junit4]   2> 2640 T25 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 2642 T25 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2642 T25 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2643 T25 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2643 T25 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2644 T25 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2645 T25 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2645 T25 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2646 T25 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2646 T25 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2647 T25 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2648 T25 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2648 T25 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2665 T54 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@29578426 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2665 T25 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2666 T25 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2667 T25 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2668 T25 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2680 T25 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2683 T25 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:32584
[junit4:junit4]   2> 2685 T25 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2685 T25 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2686 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2724 T25 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2725 T25 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2726 T25 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2726 T25 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2727 T25 oasc.CoreContainer.<init> New CoreContainer 1083851990
[junit4:junit4]   2> 2728 T25 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2728 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2758 T25 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2781 T25 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2782 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2784 T25 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2785 T25 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2831 T25 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2886 T25 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 2887 T25 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2910 T25 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 3707 T25 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 3720 T25 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 3723 T25 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 3739 T25 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3745 T25 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3748 T25 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3750 T25 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3752 T25 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351665123359/shard1/data/
[junit4:junit4]   2> 3753 T25 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 3754 T25 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351665123359/shard1/data/index/
[junit4:junit4]   2> 3756 T25 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351665123359/shard1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 3757 T25 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351665123359/shard1/data/index forceNew:false
[junit4:junit4]   2> 3761 T25 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3762 T25 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3763 T25 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 3764 T25 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 3765 T25 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 3765 T25 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 3771 T25 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 3776 T25 oass.SolrIndexSearcher.<init> Opening Searcher@5a7691c0 main
[junit4:junit4]   2> 3776 T25 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3777 T25 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 3777 T25 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 3778 T25 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 3778 T25 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3779 T25 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 3779 T25 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3779 T25 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3780 T25 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3780 T25 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3781 T25 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3781 T25 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3781 T25 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 3793 T25 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 3793 T68 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a7691c0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 3794 T25 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3795 T25 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3795 T25 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3804 T25 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 3806 T25 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:16512
[junit4:junit4]   2> 3807 T25 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3808 T25 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 3808 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3830 T25 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 3831 T25 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3831 T25 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 3832 T25 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 3832 T25 oasc.CoreContainer.<init> New CoreContainer 134585789
[junit4:junit4]   2> 3833 T25 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3833 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3854 T25 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3875 T25 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 3876 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 3877 T25 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 3877 T25 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 3909 T25 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 3947 T25 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 3948 T25 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 3969 T25 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 4434 T25 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 4447 T25 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 4451 T25 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 4469 T25 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4474 T25 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4478 T25 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4480 T25 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4481 T25 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.PeerSyncTest-1351665123359/shard2/data/
[junit4:junit4]   2> 4482 T25 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 4483 T25 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.PeerSyncTest-1351665123359/shard2/data/index/
[junit4:junit4]   2> 4484 T25 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.PeerSyncTest-1351665123359/shard2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 4485 T25 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.update.PeerSyncTest-1351665123359/shard2/data/index forceNew:false
[junit4:junit4]   2> 4488 T25 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@5b489c06 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3755e508; maxCacheMB=0.9931640625 maxMergeSizeMB=0.662109375)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4489 T25 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4490 T25 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 4491 T25 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 4492 T25 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 4492 T25 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 4498 T25 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 4504 T25 oass.SolrIndexSearcher.<init> Opening Searcher@160bc7c0 main
[junit4:junit4]   2> 4505 T25 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4506 T25 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 4506 T25 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 4507 T25 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 4507 T25 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4507 T25 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 4508 T25 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4508 T25 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4509 T25 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4509 T25 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4509 T25 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4510 T25 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4510 T25 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 4522 T25 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 4522 T80 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@160bc7c0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 4522 T25 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 4523 T25 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 4524 T25 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@3351e824
[junit4:junit4]   2> 5600 T52 C5 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5602 T52 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5676 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=115 
[junit4:junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@aede59e
[junit4:junit4]   2> 5905 T65 C6 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 5920 T65 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:29678/solr] nUpdates=100
[junit4:junit4]   2> 5927 T65 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE.  We have no versions.  sync failed.
[junit4:junit4]   2> 5928 T65 C6 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:29678/solr&version=2&getVersions=100} status=0 QTime=67 
[junit4:junit4]   2> 5938 T67 C6 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5938 T67 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5952 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=17 
[junit4:junit4]   2> 5961 T65 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:29678/solr] nUpdates=100
[junit4:junit4]   2> 5974 T53 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=15 
[junit4:junit4]   2> 5974 T51 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=11 
[junit4:junit4]   2> 5975 T65 C6 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 1 versions from 127.0.0.1:29678/solr
[junit4:junit4]   2> 5976 T65 C6 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Our versions are newer. ourLowThreshold=1 otherHigh=1
[junit4:junit4]   2> 5977 T65 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 5978 T65 C6 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:29678/solr&version=2&getVersions=100} status=0 QTime=17 
[junit4:junit4]   2> 5986 T52 C5 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6000 T52 C5 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Lucene41_0.doc, _0_Lucene41_0.pos, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0.fdt, _0_Lucene41_0.tip]
[junit4:junit4]   2> 6001 T52 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 6004 T52 C5 oass.SolrIndexSearcher.<init> Opening Searcher@517c804b main
[junit4:junit4]   2> 6004 T52 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6006 T54 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@517c804b main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 6006 T52 C5 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6006 T52 C5 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 20
[junit4:junit4]   2> 6054 T66 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6058 T66 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Lucene41_0.doc, _0_Lucene41_0.pos, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0.fdt, _0_Lucene41_0.tip]
[junit4:junit4]   2> 6058 T66 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 6061 T66 C6 oass.SolrIndexSearcher.<init> Opening Searcher@1471cb25 main
[junit4:junit4]   2> 6061 T66 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6062 T68 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1471cb25 main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 6062 T66 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6063 T66 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 9
[junit4:junit4]   2> 6081 T51 C5 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 6099 T67 C6 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 6118 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=8 
[junit4:junit4]   2> 6122 T65 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:29678/solr] nUpdates=100
[junit4:junit4]   2> 6126 T52 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 6127 T65 C6 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 2 versions from 127.0.0.1:29678/solr
[junit4:junit4]   2> 6128 T65 C6 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:29678/solrn=1 versions=[2]
[junit4:junit4]   2> 6132 T51 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=2&qt=/get&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6140 T65 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 6141 T65 C6 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:29678/solr&version=2&getVersions=100} status=0 QTime=20 
[junit4:junit4]   2> 6145 T53 C5 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6203 T53 C5 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Lucene41_0.doc, _0_Lucene41_0.pos, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0.fdt, _0_Lucene41_0.tip]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_3,generation=3,filenames=[_1_nrm.cfs, _1_Lucene41_0.doc, _0.fnm, _0_Lucene41_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1.fnm, _1_nrm.cfe, _1.fdx, _1_Lucene41_0.pos, _1.fdt, _0_Lucene41_0.doc, _1.si, _0_nrm.cfe, _0.fdx, _0.si, _0_Lucene41_0.tim, segments_3, _0_Lucene41_0.tip, _0.fdt]
[junit4:junit4]   2> 6204 T53 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 6214 T53 C5 oass.SolrIndexSearcher.<init> Opening Searcher@59243f75 main
[junit4:junit4]   2> 6215 T53 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6216 T54 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@59243f75 main{StandardDirectoryReader(segments_3:5 _0(4.1):C1 _1(4.1):C1)}
[junit4:junit4]   2> 6216 T53 C5 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6217 T53 C5 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 72
[junit4:junit4]   2> 6223 T65 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6236 T65 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_2,generation=2,filenames=[_0.fnm, _0_Lucene41_0.doc, _0_Lucene41_0.pos, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0.fdt, _0_Lucene41_0.tip]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_3,generation=3,filenames=[_1_nrm.cfs, _1_Lucene41_0.doc, _0.fnm, _0_Lucene41_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1.fnm, _1_nrm.cfe, _1.fdx, _1_Lucene41_0.pos, _1.fdt, _0_Lucene41_0.doc, _1.si, _0_nrm.cfe, _0.fdx, _0.si, _0_Lucene41_0.tim, segments_3, _0_Lucene41_0.tip, _0.fdt]
[junit4:junit4]   2> 6237 T65 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 6245 T65 C6 oass.SolrIndexSearcher.<init> Opening Searcher@2eb481ba main
[junit4:junit4]   2> 6246 T65 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6247 T68 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2eb481ba main{StandardDirectoryReader(segments_3:5 _0(4.1):C1 _1(4.1):C1)}
[junit4:junit4]   2> 6247 T65 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6248 T65 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 26
[junit4:junit4]   2> 6252 T51 C5 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 6257 T66 C6 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=2 status=0 QTime=0 
[junit4:junit4]   2> 6266 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 6274 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 6284 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 6292 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6302 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 6313 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=6 
[junit4:junit4]   2> 6323 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 6334 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=6 
[junit4:junit4]   2> 6338 T65 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:29678/solr] nUpdates=100
[junit4:junit4]   2> 6342 T51 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=0 
[junit4:junit4]   2> 6344 T65 C6 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 10 versions from 127.0.0.1:29678/solr
[junit4:junit4]   2> 6344 T65 C6 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:29678/solrn=8 versions=[10, 9, 8, 7, 6, 5, 4, 3]
[junit4:junit4]   2> 6349 T53 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=10,9,8,7,6,5,4,3&qt=/get&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6383 T65 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 6383 T65 C6 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:29678/solr&version=2&getVersions=100} status=0 QTime=46 
[junit4:junit4]   2> 6387 T51 C5 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6415 T51 C5 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_3,generation=3,filenames=[_1_nrm.cfs, _1_Lucene41_0.doc, _0.fnm, _0_Lucene41_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1.fnm, _1_nrm.cfe, _1.fdx, _1_Lucene41_0.pos, _1.fdt, _0_Lucene41_0.doc, _1.si, _0_nrm.cfe, _0.fdx, _0.si, _0_Lucene41_0.tim, segments_3, _0_Lucene41_0.tip, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_4,generation=4,filenames=[_2_Lucene41_0.pos, _2.si, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _0_Lucene41_0.pos, _2_Lucene41_0.tip, _1.fnm, _2_nrm.cfe, _2_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1_nrm.cfs, _0.fnm, _1_Lucene41_0.tim, _2.fdt, _1_Lucene41_0.tip, _2_Lucene41_0.doc, _0_nrm.cfs, _2.fdx, _2.fnm, _1_nrm.cfe, _1_Lucene41_0.pos, _1.fdx, _0_Lucene41_0.doc, _1.fdt, _1.si, _0_nrm.cfe, _0.fdx, _0.fdt, segments_4]
[junit4:junit4]   2> 6416 T51 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 6422 T51 C5 oass.SolrIndexSearcher.<init> Opening Searcher@27daf835 main
[junit4:junit4]   2> 6423 T51 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6424 T54 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@27daf835 main{StandardDirectoryReader(segments_4:7 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8)}
[junit4:junit4]   2> 6424 T51 C5 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6426 T51 C5 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 39
[junit4:junit4]   2> 6430 T67 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6459 T67 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_3,generation=3,filenames=[_1_nrm.cfs, _1_Lucene41_0.doc, _0.fnm, _0_Lucene41_0.pos, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1.fnm, _1_nrm.cfe, _1.fdx, _1_Lucene41_0.pos, _1.fdt, _0_Lucene41_0.doc, _1.si, _0_nrm.cfe, _0.fdx, _0.si, _0_Lucene41_0.tim, segments_3, _0_Lucene41_0.tip, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_4,generation=4,filenames=[_2_Lucene41_0.pos, _2.si, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _0_Lucene41_0.pos, _2_Lucene41_0.tip, _1.fnm, _2_nrm.cfe, _2_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1_nrm.cfs, _0.fnm, _1_Lucene41_0.tim, _2.fdt, _1_Lucene41_0.tip, _2_Lucene41_0.doc, _0_nrm.cfs, _2.fdx, _2.fnm, _1_nrm.cfe, _1_Lucene41_0.pos, _1.fdx, _0_Lucene41_0.doc, _1.fdt, _1.si, _0_nrm.cfe, _0.fdx, _0.fdt, segments_4]
[junit4:junit4]   2> 6459 T67 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 6466 T67 C6 oass.SolrIndexSearcher.<init> Opening Searcher@36ed5ba6 main
[junit4:junit4]   2> 6466 T67 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6468 T68 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@36ed5ba6 main{StandardDirectoryReader(segments_4:7 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8)}
[junit4:junit4]   2> 6468 T67 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6469 T67 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 38
[junit4:junit4]   2> 6473 T52 C5 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=10 status=0 QTime=1 
[junit4:junit4]   2> 6480 T66 C6 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=10 status=0 QTime=1 
[junit4:junit4]   2> 6493 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6501 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6507 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6512 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6517 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6522 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6527 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6532 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6537 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6542 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6547 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6552 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6556 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6561 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6565 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6570 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6576 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6581 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6586 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6591 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6597 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6602 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6607 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6614 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6620 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6627 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6633 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6641 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6647 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6652 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6658 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6664 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6669 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6675 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6680 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6687 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6693 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6699 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6704 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6711 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6719 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6725 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6732 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6737 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6744 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6752 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6758 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6765 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6771 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6777 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6785 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 6791 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6797 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6803 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6815 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6822 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6827 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6832 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6837 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6842 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6848 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6853 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6858 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6863 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6869 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6874 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6882 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6887 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6892 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6897 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6903 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6908 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6913 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6918 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6923 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6929 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6934 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6939 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6944 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6949 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6955 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6960 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6965 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6970 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6975 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 6980 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6985 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6990 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 6995 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7000 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7005 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7010 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7015 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7020 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7025 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7029 T67 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:29678/solr] nUpdates=100
[junit4:junit4]   2> 7034 T52 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 7035 T67 C6 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:29678/solr
[junit4:junit4]   2> 7036 T67 C6 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Our versions are too old. ourHighThreshold=8 otherLowThreshold=25
[junit4:junit4]   2> 7036 T67 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync failed
[junit4:junit4]   2> 7036 T67 C6 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:29678/solr&version=2&getVersions=100} status=0 QTime=8 
[junit4:junit4]   2> 7042 T65 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7048 T66 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7052 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7057 T65 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7062 T66 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7067 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7071 T65 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7076 T66 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7080 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7084 T65 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7089 T66 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7093 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7098 T66 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7101 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7105 T65 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7110 T66 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7114 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7118 T65 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7123 T66 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7127 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7131 T65 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7136 T66 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7139 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7143 T65 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7148 T66 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7151 T67 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:29678/solr] nUpdates=100
[junit4:junit4]   2> 7155 T53 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 7156 T67 C6 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:29678/solr
[junit4:junit4]   2> 7156 T67 C6 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:29678/solrn=70 versions=[105, 104, 103, 102, 101, 100, 99, 98, 97, 96, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 79, 78, 77, 76, 75, 74, 73, 72, 71, 70, 69, 68, 67, 66, 65, 64, 63, 62, 61, 60, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 43, 42, 41, 40, 39, 38, 37, 36]
[junit4:junit4]   2> 7165 T52 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=105,104,103,102,101,100,99,98,97,96,95,94,93,92,91,90,89,88,87,86,85,84,83,82,81,80,79,78,77,76,75,74,73,72,71,70,69,68,67,66,65,64,63,62,61,60,59,58,57,56,55,54,53,52,51,50,49,48,47,46,45,44,43,42,41,40,39,38,37,36&qt=/get&wt=javabin&version=2} status=0 QTime=5 
[junit4:junit4]   2> 7222 T67 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 7223 T67 C6 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:29678/solr&version=2&getVersions=100} status=0 QTime=72 
[junit4:junit4]   2> 7227 T53 C5 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 7250 T53 C5 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_4,generation=4,filenames=[_2_Lucene41_0.pos, _2.si, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _0_Lucene41_0.pos, _2_Lucene41_0.tip, _1.fnm, _2_nrm.cfe, _2_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1_nrm.cfs, _0.fnm, _1_Lucene41_0.tim, _2.fdt, _1_Lucene41_0.tip, _2_Lucene41_0.doc, _0_nrm.cfs, _2.fdx, _2.fnm, _1_nrm.cfe, _1_Lucene41_0.pos, _1.fdx, _0_Lucene41_0.doc, _1.fdt, _1.si, _0_nrm.cfe, _0.fdx, _0.fdt, segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_5,generation=5,filenames=[_3_nrm.cfe, _2_Lucene41_0.pos, _2.si, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _0_Lucene41_0.pos, _3.fdt, _3.fnm, _2_Lucene41_0.tip, _1.fnm, _2_nrm.cfe, _3_Lucene41_0.pos, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _2_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1_nrm.cfs, _0.fnm, _1_Lucene41_0.tim, _2.fdt, _1_Lucene41_0.tip, _2_Lucene41_0.doc, _0_nrm.cfs, _2.fdx, _2.fnm, _1_nrm.cfe, _3.fdx, _3_nrm.cfs, _1_Lucene41_0.pos, _1.fdx, _3_Lucene41_0.doc, _0_Lucene41_0.doc, _1.fdt, _1.si, _0_nrm.cfe, _3.si, _0.fdx, segments_5, _0.fdt]
[junit4:junit4]   2> 7251 T53 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 5
[junit4:junit4]   2> 7255 T53 C5 oass.SolrIndexSearcher.<init> Opening Searcher@7439aca7 main
[junit4:junit4]   2> 7256 T53 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7256 T54 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7439aca7 main{StandardDirectoryReader(segments_5:9 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95)}
[junit4:junit4]   2> 7257 T53 C5 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 7258 T53 C5 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 31
[junit4:junit4]   2> 7262 T65 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 7283 T65 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_4,generation=4,filenames=[_2_Lucene41_0.pos, _2.si, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _0_Lucene41_0.pos, _2_Lucene41_0.tip, _1.fnm, _2_nrm.cfe, _2_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1_nrm.cfs, _0.fnm, _1_Lucene41_0.tim, _2.fdt, _1_Lucene41_0.tip, _2_Lucene41_0.doc, _0_nrm.cfs, _2.fdx, _2.fnm, _1_nrm.cfe, _1_Lucene41_0.pos, _1.fdx, _0_Lucene41_0.doc, _1.fdt, _1.si, _0_nrm.cfe, _0.fdx, _0.fdt, segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_5,generation=5,filenames=[_3_nrm.cfe, _2_Lucene41_0.pos, _2.si, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _0_Lucene41_0.pos, _3.fdt, _3.fnm, _2_Lucene41_0.tip, _1.fnm, _2_nrm.cfe, _3_Lucene41_0.pos, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _2_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1_nrm.cfs, _0.fnm, _1_Lucene41_0.tim, _2.fdt, _1_Lucene41_0.tip, _2_Lucene41_0.doc, _0_nrm.cfs, _2.fdx, _2.fnm, _1_nrm.cfe, _3.fdx, _3_nrm.cfs, _1_Lucene41_0.pos, _1.fdx, _3_Lucene41_0.doc, _0_Lucene41_0.doc, _1.fdt, _1.si, _0_nrm.cfe, _3.si, _0.fdx, segments_5, _0.fdt]
[junit4:junit4]   2> 7283 T65 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 5
[junit4:junit4]   2> 7288 T65 C6 oass.SolrIndexSearcher.<init> Opening Searcher@79fc7299 main
[junit4:junit4]   2> 7288 T65 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7289 T68 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79fc7299 main{StandardDirectoryReader(segments_5:9 _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95)}
[junit4:junit4]   2> 7289 T65 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 7290 T65 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 29
[junit4:junit4]   2> 7463 T51 C5 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=105 status=0 QTime=170 
[junit4:junit4]   2> 7472 T66 C6 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=105 status=0 QTime=3 
[junit4:junit4]   2> 7481 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 7486 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7614 T51 C5 oass.SolrIndexSearcher.<init> Opening Searcher@461fbe88 realtime
[junit4:junit4]   2> 7615 T51 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7616 T51 C5 REQ [collection1] webapp=/solr path=/update params={_version_=-1003&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=126 
[junit4:junit4]   2> 7622 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7627 T52 C5 REQ [collection1] webapp=/solr path=/update params={_version_=-1005&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7630 T65 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:29678/solr] nUpdates=100
[junit4:junit4]   2> 7639 T53 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=5 
[junit4:junit4]   2> 7640 T65 C6 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:29678/solr
[junit4:junit4]   2> 7641 T65 C6 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:29678/solrn=5 versions=[-1005, 1004, -1003, 1002, 1001]
[junit4:junit4]   2> 7645 T52 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=-1005,1004,-1003,1002,1001&qt=/get&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7666 T65 C6 oass.SolrIndexSearcher.<init> Opening Searcher@12627bc7 realtime
[junit4:junit4]   2> 7667 T65 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7670 T65 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 7671 T65 C6 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:29678/solr&version=2&getVersions=100} status=0 QTime=41 
[junit4:junit4]   2> 7675 T51 C5 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 7680 T51 C5 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_5,generation=5,filenames=[_3_nrm.cfe, _2_Lucene41_0.pos, _2.si, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _0_Lucene41_0.pos, _3.fdt, _3.fnm, _2_Lucene41_0.tip, _1.fnm, _2_nrm.cfe, _3_Lucene41_0.pos, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _2_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1_nrm.cfs, _0.fnm, _1_Lucene41_0.tim, _2.fdt, _1_Lucene41_0.tip, _2_Lucene41_0.doc, _0_nrm.cfs, _2.fdx, _2.fnm, _1_nrm.cfe, _3.fdx, _3_nrm.cfs, _1_Lucene41_0.pos, _1.fdx, _3_Lucene41_0.doc, _0_Lucene41_0.doc, _1.fdt, _1.si, _0_nrm.cfe, _3.si, _0.fdx, segments_5, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_6,generation=6,filenames=[_2_Lucene41_0.pos, _3_nrm.cfe, _5_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_Lucene41_0.tim, _0_Lucene41_0.pos, _5_Lucene41_0.tip, _5_Lucene41_0.pos, _3.fdt, _2_Lucene41_0.tip, _1.fnm, _3_Lucene41_0.pos, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fnm, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _5.si, _1.fdx, _1_Lucene41_0.pos, _3_nrm.cfs, _3_Lucene41_0.doc, _1.fdt, _0_Lucene41_0.doc, _5_nrm.cfs, _3.si, _2.si, _3.fnm, _2_nrm.cfe, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _2_nrm.cfs, _1_nrm.cfs, _5.fnm, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1_nrm.cfe, _2.fnm, _3.fdx, _1.si, _0_nrm.cfe, _5.fdx, _0.fdx, segments_6, _0.fdt]
[junit4:junit4]   2> 7681 T51 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 6
[junit4:junit4]   2> 7683 T51 C5 oass.SolrIndexSearcher.<init> Opening Searcher@ba0abea main
[junit4:junit4]   2> 7683 T51 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7685 T54 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ba0abea main{StandardDirectoryReader(segments_6:16:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1)}
[junit4:junit4]   2> 7685 T51 C5 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 7687 T51 C5 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 12
[junit4:junit4]   2> 7692 T66 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 7712 T66 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_5,generation=5,filenames=[_3_nrm.cfe, _2_Lucene41_0.pos, _2.si, _1_Lucene41_0.doc, _2_Lucene41_0.tim, _0_Lucene41_0.pos, _3.fdt, _3.fnm, _2_Lucene41_0.tip, _1.fnm, _2_nrm.cfe, _3_Lucene41_0.pos, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _2_nrm.cfs, _0.si, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _1_nrm.cfs, _0.fnm, _1_Lucene41_0.tim, _2.fdt, _1_Lucene41_0.tip, _2_Lucene41_0.doc, _0_nrm.cfs, _2.fdx, _2.fnm, _1_nrm.cfe, _3.fdx, _3_nrm.cfs, _1_Lucene41_0.pos, _1.fdx, _3_Lucene41_0.doc, _0_Lucene41_0.doc, _1.fdt, _1.si, _0_nrm.cfe, _3.si, _0.fdx, segments_5, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_6,generation=6,filenames=[_2_Lucene41_0.pos, _3_nrm.cfe, _5_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_Lucene41_0.tim, _0_Lucene41_0.pos, _5_Lucene41_0.tip, _5_Lucene41_0.pos, _3.fdt, _2_Lucene41_0.tip, _1.fnm, _3_Lucene41_0.pos, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fnm, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _5.si, _1.fdx, _1_Lucene41_0.pos, _3_nrm.cfs, _3_Lucene41_0.doc, _1.fdt, _0_Lucene41_0.doc, _5_nrm.cfs, _3.si, _2.si, _3.fnm, _2_nrm.cfe, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _2_nrm.cfs, _1_nrm.cfs, _5.fnm, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1_nrm.cfe, _2.fnm, _3.fdx, _1.si, _0_nrm.cfe, _5.fdx, _0.fdx, segments_6, _0.fdt]
[junit4:junit4]   2> 7713 T66 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 6
[junit4:junit4]   2> 7717 T66 C6 oass.SolrIndexSearcher.<init> Opening Searcher@402c3549 main
[junit4:junit4]   2> 7718 T66 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7719 T68 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@402c3549 main{StandardDirectoryReader(segments_6:16:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1)}
[junit4:junit4]   2> 7720 T66 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 7772 T66 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 80
[junit4:junit4]   2> 7779 T52 C5 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=106 status=0 QTime=3 
[junit4:junit4]   2> 7786 T67 C6 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=106 status=0 QTime=2 
[junit4:junit4]   2> 7795 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 7800 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7814 T52 C5 oass.SolrIndexSearcher.<init> Opening Searcher@6064af0f realtime
[junit4:junit4]   2> 7814 T52 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7815 T52 C5 REQ [collection1] webapp=/solr path=/update params={_version_=-2003&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=11 
[junit4:junit4]   2> 7821 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7826 T53 C5 REQ [collection1] webapp=/solr path=/update params={_version_=-2005&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7833 T65 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 7847 T66 C6 oass.SolrIndexSearcher.<init> Opening Searcher@e28099 realtime
[junit4:junit4]   2> 7847 T66 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7848 T66 C6 REQ [collection1] webapp=/solr path=/update params={_version_=-2003&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=10 
[junit4:junit4]   2> 7854 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7859 T65 C6 REQ [collection1] webapp=/solr path=/update params={_version_=-2005&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7862 T66 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:29678/solr] nUpdates=100
[junit4:junit4]   2> 7868 T52 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 7869 T66 C6 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:29678/solr
[junit4:junit4]   2> 7869 T66 C6 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:29678/solrn=1 versions=[2002]
[junit4:junit4]   2> 7873 T51 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=2002&qt=/get&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7875 T66 C6 oasu.DirectUpdateHandler2.addDoc Reordered DBQs detected.  Update=add{flags=12,_version_=2002,id=2001} DBQs=[DBQ{version=2003,q=id:2001 OR id:2002}]
[junit4:junit4]   2> 7886 T66 C6 oass.SolrIndexSearcher.<init> Opening Searcher@2543472c realtime
[junit4:junit4]   2> 7886 T66 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7895 T66 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 7895 T66 C6 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:29678/solr&version=2&getVersions=100} status=0 QTime=33 
[junit4:junit4]   2> 7899 T53 C5 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 7906 T53 C5 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_6,generation=6,filenames=[_2_Lucene41_0.pos, _3_nrm.cfe, _5_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_Lucene41_0.tim, _0_Lucene41_0.pos, _5_Lucene41_0.tip, _5_Lucene41_0.pos, _3.fdt, _2_Lucene41_0.tip, _1.fnm, _3_Lucene41_0.pos, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fnm, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _5.si, _1.fdx, _1_Lucene41_0.pos, _3_nrm.cfs, _3_Lucene41_0.doc, _1.fdt, _0_Lucene41_0.doc, _5_nrm.cfs, _3.si, _2.si, _3.fnm, _2_nrm.cfe, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _2_nrm.cfs, _1_nrm.cfs, _5.fnm, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1_nrm.cfe, _2.fnm, _3.fdx, _1.si, _0_nrm.cfe, _5.fdx, _0.fdx, segments_6, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_7,generation=7,filenames=[_2_Lucene41_0.pos, _3_nrm.cfe, _5_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_Lucene41_0.tim, _0_Lucene41_0.pos, _5_Lucene41_0.tip, _5_Lucene41_0.pos, _7.si, _3.fdt, _2_Lucene41_0.tip, _1.fnm, _7.fnm, _3_Lucene41_0.pos, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fnm, _2.fdt, _7.fdx, _2_Lucene41_0.doc, _2.fdx, _7.fdt, _5.si, _1.fdx, _1_Lucene41_0.pos, _3_nrm.cfs, _3_Lucene41_0.doc, _1.fdt, _0_Lucene41_0.doc, _5_nrm.cfs, _3.si, _2.si, _7_Lucene41_0.pos, _3.fnm, _2_nrm.cfe, _7_nrm.cfs, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _2_nrm.cfs, _7_Lucene41_0.doc, _1_nrm.cfs, _5.fnm, _7_nrm.cfe, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1_nrm.cfe, _2.fnm, _3.fdx, segments_7, _1.si, _0_nrm.cfe, _7_Lucene41_0.tim, _5.fdx, _0.fdx, _7_Lucene41_0.tip, _0.fdt]
[junit4:junit4]   2> 7907 T53 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 7
[junit4:junit4]   2> 7910 T53 C5 oass.SolrIndexSearcher.<init> Opening Searcher@4b469bea main
[junit4:junit4]   2> 7910 T53 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7912 T54 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4b469bea main{StandardDirectoryReader(segments_7:23:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1 _7(4.1):C1)}
[junit4:junit4]   2> 7912 T53 C5 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 7912 T53 C5 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 13
[junit4:junit4]   2> 7917 T65 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 7921 T65 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_6,generation=6,filenames=[_2_Lucene41_0.pos, _3_nrm.cfe, _5_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_Lucene41_0.tim, _0_Lucene41_0.pos, _5_Lucene41_0.tip, _5_Lucene41_0.pos, _3.fdt, _2_Lucene41_0.tip, _1.fnm, _3_Lucene41_0.pos, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fnm, _2.fdt, _2_Lucene41_0.doc, _2.fdx, _5.si, _1.fdx, _1_Lucene41_0.pos, _3_nrm.cfs, _3_Lucene41_0.doc, _1.fdt, _0_Lucene41_0.doc, _5_nrm.cfs, _3.si, _2.si, _3.fnm, _2_nrm.cfe, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _2_nrm.cfs, _1_nrm.cfs, _5.fnm, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1_nrm.cfe, _2.fnm, _3.fdx, _1.si, _0_nrm.cfe, _5.fdx, _0.fdx, segments_6, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_7,generation=7,filenames=[_7_1.del, _2_Lucene41_0.pos, _3_nrm.cfe, _5_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_Lucene41_0.tim, _0_Lucene41_0.pos, _5_Lucene41_0.tip, _5_Lucene41_0.pos, _7.si, _3.fdt, _2_Lucene41_0.tip, _1.fnm, _7.fnm, _3_Lucene41_0.pos, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fnm, _2.fdt, _7.fdx, _2_Lucene41_0.doc, _2.fdx, _7.fdt, _5.si, _1.fdx, _1_Lucene41_0.pos, _3_nrm.cfs, _3_Lucene41_0.doc, _1.fdt, _0_Lucene41_0.doc, _5_nrm.cfs, _3.si, _2.si, _7_Lucene41_0.pos, _3.fnm, _2_nrm.cfe, _7_nrm.cfs, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _2_nrm.cfs, _7_Lucene41_0.doc, _1_nrm.cfs, _5.fnm, _7_nrm.cfe, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1_nrm.cfe, _2.fnm, _3.fdx, segments_7, _1.si, _0_nrm.cfe, _7_Lucene41_0.tim, _5.fdx, _0.fdx, _7_Lucene41_0.tip, _0.fdt]
[junit4:junit4]   2> 7922 T65 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 7
[junit4:junit4]   2> 7923 T65 C6 oass.SolrIndexSearcher.<init> Opening Searcher@785b38d6 main
[junit4:junit4]   2> 7924 T65 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 7926 T68 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@785b38d6 main{StandardDirectoryReader(segments_6:22:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1 _7(4.1):C2/1)}
[junit4:junit4]   2> 7926 T65 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 7926 T65 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 9
[junit4:junit4]   2> 7931 T52 C5 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=107 status=0 QTime=1 
[junit4:junit4]   2> 7938 T66 C6 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=107 status=0 QTime=1 
[junit4:junit4]   2> 7948 T51 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 7952 T67 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7955 T53 C5 REQ [collection1] webapp=/solr path=/update params={_version_=3000&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 7959 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7963 T51 C5 REQ [collection1] webapp=/solr path=/update params={_version_=3001&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7967 T65 C6 REQ [collection1] webapp=/solr path=/update params={_version_=3001&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7971 T53 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7975 T52 C5 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7981 T66 C6 REQ [collection1] webapp=/solr path=/update params={update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 7983 T67 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= START replicas=[127.0.0.1:29678/solr] nUpdates=100
[junit4:junit4]   2> 7987 T53 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=1 
[junit4:junit4]   2> 7988 T67 C6 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=  Received 100 versions from 127.0.0.1:29678/solr
[junit4:junit4]   2> 7988 T67 C6 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url= Requesting updates from 127.0.0.1:29678/solrn=2 versions=[3004, 3003]
[junit4:junit4]   2> 7991 T51 C5 REQ [collection1] webapp=/solr path=/get params={distrib=false&getUpdates=3004,3003&qt=/get&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 7993 T67 C6 oasu.PeerSync.sync PeerSync: core=collection1 url= DONE. sync succeeded
[junit4:junit4]   2> 7994 T67 C6 REQ [collection1] webapp=/solr path=/get params={qt=/get&wt=javabin&sync=127.0.0.1:29678/solr&version=2&getVersions=100} status=0 QTime=11 
[junit4:junit4]   2> 7998 T52 C5 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 8004 T52 C5 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_7,generation=7,filenames=[_2_Lucene41_0.pos, _3_nrm.cfe, _5_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_Lucene41_0.tim, _0_Lucene41_0.pos, _5_Lucene41_0.tip, _5_Lucene41_0.pos, _7.si, _3.fdt, _2_Lucene41_0.tip, _1.fnm, _7.fnm, _3_Lucene41_0.pos, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fnm, _2.fdt, _7.fdx, _2_Lucene41_0.doc, _2.fdx, _7.fdt, _5.si, _1.fdx, _1_Lucene41_0.pos, _3_nrm.cfs, _3_Lucene41_0.doc, _1.fdt, _0_Lucene41_0.doc, _5_nrm.cfs, _3.si, _2.si, _7_Lucene41_0.pos, _3.fnm, _2_nrm.cfe, _7_nrm.cfs, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _2_nrm.cfs, _7_Lucene41_0.doc, _1_nrm.cfs, _5.fnm, _7_nrm.cfe, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1_nrm.cfe, _2.fnm, _3.fdx, segments_7, _1.si, _0_nrm.cfe, _7_Lucene41_0.tim, _5.fdx, _0.fdx, _7_Lucene41_0.tip, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7cd76237 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6d69c9a2),segFN=segments_8,generation=8,filenames=[_2_Lucene41_0.pos, _3_nrm.cfe, _5_nrm.cfe, _8_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_Lucene41_0.tim, _0_Lucene41_0.pos, _5_Lucene41_0.tip, _5_Lucene41_0.pos, _7.si, _3.fdt, _2_Lucene41_0.tip, _1.fnm, _8_1.del, _8.si, _7.fnm, _3_Lucene41_0.pos, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fnm, _2.fdt, _7.fdx, _2_Lucene41_0.doc, _2.fdx, _7.fdt, _8_nrm.cfs, _5.si, _8_Lucene41_0.doc, _1.fdx, _1_Lucene41_0.pos, _3_nrm.cfs, _8.fnm, _3_Lucene41_0.doc, _1.fdt, _0_Lucene41_0.doc, _5_nrm.cfs, _3.si, _2.si, _7_Lucene41_0.pos, _3.fnm, _2_nrm.cfe, _8_Lucene41_0.pos, _7_nrm.cfs, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _2_nrm.cfs, _7_Lucene41_0.doc, _1_nrm.cfs, _5.fnm, _7_nrm.cfe, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1_nrm.cfe, _2.fnm, _3.fdx, segments_8, _1.si, _0_nrm.cfe, _7_Lucene41_0.tim, _5.fdx, _0.fdx, _8.fdx, _8_Lucene41_0.tim, _7_Lucene41_0.tip, _0.fdt, _8.fdt, _8_Lucene41_0.tip]
[junit4:junit4]   2> 8005 T52 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 8
[junit4:junit4]   2> 8008 T52 C5 oass.SolrIndexSearcher.<init> Opening Searcher@73974028 main
[junit4:junit4]   2> 8008 T52 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 8009 T54 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73974028 main{StandardDirectoryReader(segments_8:25:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1 _7(4.1):C1 _8(4.1):C4/1)}
[junit4:junit4]   2> 8009 T52 C5 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 8010 T52 C5 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 12
[junit4:junit4]   2> 8014 T65 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 8022 T65 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_7,generation=7,filenames=[_7_1.del, _2_Lucene41_0.pos, _3_nrm.cfe, _5_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_Lucene41_0.tim, _0_Lucene41_0.pos, _5_Lucene41_0.tip, _5_Lucene41_0.pos, _7.si, _3.fdt, _2_Lucene41_0.tip, _1.fnm, _7.fnm, _3_Lucene41_0.pos, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fnm, _2.fdt, _7.fdx, _2_Lucene41_0.doc, _2.fdx, _7.fdt, _5.si, _1.fdx, _1_Lucene41_0.pos, _3_nrm.cfs, _3_Lucene41_0.doc, _1.fdt, _0_Lucene41_0.doc, _5_nrm.cfs, _3.si, _2.si, _7_Lucene41_0.pos, _3.fnm, _2_nrm.cfe, _7_nrm.cfs, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _2_nrm.cfs, _7_Lucene41_0.doc, _1_nrm.cfs, _5.fnm, _7_nrm.cfe, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1_nrm.cfe, _2.fnm, _3.fdx, segments_7, _1.si, _0_nrm.cfe, _7_Lucene41_0.tim, _5.fdx, _0.fdx, _7_Lucene41_0.tip, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3c05b84b lockFactory=org.apache.lucene.store.NativeFSLockFactory@13baa8d8),segFN=segments_8,generation=8,filenames=[_7_1.del, _2_Lucene41_0.pos, _3_nrm.cfe, _5_nrm.cfe, _8_nrm.cfe, _1_Lucene41_0.doc, _5.fdt, _2_Lucene41_0.tim, _5_Lucene41_0.tim, _0_Lucene41_0.pos, _5_Lucene41_0.tip, _5_Lucene41_0.pos, _7.si, _3.fdt, _2_Lucene41_0.tip, _1.fnm, _8.si, _7.fnm, _3_Lucene41_0.pos, _5_Lucene41_0.doc, _0_Lucene41_0.tim, _0_Lucene41_0.tip, _0.fnm, _2.fdt, _7.fdx, _2_Lucene41_0.doc, _2.fdx, _7.fdt, _8_nrm.cfs, _5.si, _8_Lucene41_0.doc, _1.fdx, _1_Lucene41_0.pos, _3_nrm.cfs, _8.fnm, _3_Lucene41_0.doc, _1.fdt, _0_Lucene41_0.doc, _5_nrm.cfs, _3.si, _2.si, _7_Lucene41_0.pos, _3.fnm, _2_nrm.cfe, _8_Lucene41_0.pos, _7_nrm.cfs, _3_Lucene41_0.tip, _3_Lucene41_0.tim, _0.si, _2_nrm.cfs, _7_Lucene41_0.doc, _1_nrm.cfs, _5.fnm, _7_nrm.cfe, _1_Lucene41_0.tim, _1_Lucene41_0.tip, _0_nrm.cfs, _1_nrm.cfe, _2.fnm, _3.fdx, segments_8, _1.si, _0_nrm.cfe, _7_Lucene41_0.tim, _5.fdx, _0.fdx, _8.fdx, _8_Lucene41_0.tim, _7_Lucene41_0.tip, _0.fdt, _8.fdt, _8_Lucene41_0.tip]
[junit4:junit4]   2> 8023 T65 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 8
[junit4:junit4]   2> 8026 T65 C6 oass.SolrIndexSearcher.<init> Opening Searcher@fce56f8 main
[junit4:junit4]   2> 8027 T65 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 8028 T68 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fce56f8 main{StandardDirectoryReader(segments_8:24:nrt _0(4.1):C1 _1(4.1):C1 _2(4.1):C8 _3(4.1):C95 _5(4.1):C1 _7(4.1):C2/1 _8(4.1):C3)}
[junit4:junit4]   2> 8028 T65 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 8029 T65 C6 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 15
[junit4:junit4]   2> 8034 T53 C5 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=110 status=0 QTime=2 
[junit4:junit4]   2> 8042 T66 C6 REQ [collection1] webapp=/solr path=/select params={sort=_version_+desc&wt=javabin&q=*:*&version=2} hits=110 status=0 QTime=2 
[junit4:junit4]   2> 8091 T25 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1064301273
[junit4:junit4]   2> 8091 T25 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@7d420fea
[junit4:junit4]   2> 8092 T25 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 8092 T25 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 8093 T25 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 8093 T25 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 8093 T25 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 8095 T25 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 8196 T25 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 8240 T25 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=294576878
[junit4:junit4]   2> 8241 T25 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3351e824
[junit4:junit4]   2> 8241 T25 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=7,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=115,cumulative_deletesById=3,cumulative_deletesByQuery=2,cumulative_errors=0}
[junit4:junit4]   2> 8242 T25 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 8242 T25 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 8242 T25 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 8244 T25 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 8292 T25 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 8393 T25 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 8437 T25 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1083851990
[junit4:junit4]   2> 8438 T25 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@aede59e
[junit4:junit4]   2> 8439 T25 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=7,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=114,cumulative_deletesById=3,cumulative_deletesByQuery=3,cumulative_errors=0}
[junit4:junit4]   2> 8439 T25 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 8440 T25 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 8440 T25 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 8443 T25 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 8542 T25 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 8643 T25 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 8688 T25 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=134585789
[junit4:junit4]   2> 8689 T25 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@76d4d81
[junit4:junit4]   2> 8689 T25 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 8690 T25 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 8690 T25 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 8691 T25 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 8691 T25 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 8692 T25 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 8792 T25 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 8807 T25 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 8810 T25 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 8817 T24 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 16 leaked thread(s).
[junit4:junit4]   2> 29068 T24 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 16 threads leaked from SUITE scope at org.apache.solr.update.PeerSyncTest: 
[junit4:junit4]   2> 	   1) Thread[id=77, name=qtp58654006-77 Acceptor3 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=46, name=qtp1755031090-46 Acceptor0 SelectChannelConnector@0.0.0.0:29678, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=35, name=qtp525030058-35 Acceptor3 SelectChannelConnector@0.0.0.0:34666, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=61, name=qtp1180675534-61 Acceptor1 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   5) Thread[id=34, name=qtp525030058-34 Acceptor2 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   6) Thread[id=32, name=qtp525030058-32 Acceptor0 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   7) Thread[id=74, name=qtp58654006-74 Acceptor0 SelectChannelConnector@0.0.0.0:16512, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   8) Thread[id=48, name=qtp1755031090-48 Acceptor2 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   9) Thread[id=75, name=qtp58654006-75 Acceptor1 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  10) Thread[id=76, name=qtp58654006-76 Acceptor2 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  11) Thread[id=60, name=qtp1180675534-60 Acceptor0 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  12) Thread[id=49, name=qtp1755031090-49 Acceptor3 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  13) Thread[id=47, name=qtp1755031090-47 Acceptor1 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  14) Thread[id=62, name=qtp1180675534-62 Acceptor2 SelectChannelConnector@0.0.0.0:32584, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  15) Thread[id=33, name=qtp525030058-33 Acceptor1 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  16) Thread[id=63, name=qtp1180675534-63 Acceptor3 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 29070 T24 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=77, name=qtp58654006-77 Acceptor3 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	   2) Thread[id=46, name=qtp1755031090-46 Acceptor0 SelectChannelConnector@0.0.0.0:29678, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	   3) Thread[id=35, name=qtp525030058-35 Acceptor3 SelectChannelConnector@0.0.0.0:34666, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	   4) Thread[id=61, name=qtp1180675534-61 Acceptor1 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	   5) Thread[id=34, name=qtp525030058-34 Acceptor2 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	   6) Thread[id=32, name=qtp525030058-32 Acceptor0 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	   7) Thread[id=74, name=qtp58654006-74 Acceptor0 SelectChannelConnector@0.0.0.0:16512, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	   8) Thread[id=48, name=qtp1755031090-48 Acceptor2 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	   9) Thread[id=75, name=qtp58654006-75 Acceptor1 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	  10) Thread[id=76, name=qtp58654006-76 Acceptor2 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	  11) Thread[id=60, name=qtp1180675534-60 Acceptor0 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	  12) Thread[id=49, name=qtp1755031090-49 Acceptor3 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	  13) Thread[id=47, name=qtp1755031090-47 Acceptor1 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	  14) Thread[id=62, name=qtp1180675534-62 Acceptor2 SelectChannelConnector@0.0.0.0:32584, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	  15) Thread[id=33, name=qtp525030058-33 Acceptor1 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	  16) Thread[id=63, name=qtp1180675534-63 Acceptor3 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 32260 T24 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=77, name=qtp58654006-77 Acceptor3 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=46, name=qtp1755031090-46 Acceptor0 SelectChannelConnector@0.0.0.0:29678, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=35, name=qtp525030058-35 Acceptor3 SelectChannelConnector@0.0.0.0:34666, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=61, name=qtp1180675534-61 Acceptor1 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   5) Thread[id=34, name=qtp525030058-34 Acceptor2 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   6) Thread[id=32, name=qtp525030058-32 Acceptor0 SelectChannelConnector@0.0.0.0:34666, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   7) Thread[id=48, name=qtp1755031090-48 Acceptor2 SelectChannelConnector@0.0.0.0:29678, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   8) Thread[id=74, name=qtp58654006-74 Acceptor0 SelectChannelConnector@0.0.0.0:16512, state=RUNNABLE, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   9) Thread[id=75, name=qtp58654006-75 Acceptor1 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  10) Thread[id=60, name=qtp1180675534-60 Acceptor0 SelectChannelConnector@0.0.0.0:32584, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	  11) Thread[id=76, name=qtp58654006-76 Acceptor2 SelectChannelConnector@0.0.0.0:16512, state=BLOCKED, group=TGRP-PeerSyncTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4

[...truncated too long message...]

       at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41: {timestamp=PostingsFormat(name=Asserting), range_facet_l=PostingsFormat(name=Asserting), id=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_sl=PostingsFormat(name=TestBloomFilteredLucene41Postings), range_facet_si=PostingsFormat(name=Asserting), teststop=PostingsFormat(name=Asserting), multiDefault=PostingsFormat(name=Asserting), intDefault=PostingsFormat(name=MockRandom)}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=en_NZ, timezone=Africa/Ndjamena
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=115844016,total=166002688
[junit4:junit4]   2> NOTE: All tests run in this JVM: [ZkControllerTest, CoreAdminHandlerTest, TestRangeQuery, UniqFieldsUpdateProcessorFactoryTest, SuggesterFSTTest, ResponseLogComponentTest, SpellCheckComponentTest, AnalysisAfterCoreReloadTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=AnalysisAfterCoreReloadTest -Dtests.seed=D4502F33E1BD685A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_NZ -Dtests.timezone=Africa/Ndjamena -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR   0.00s J0 | AnalysisAfterCoreReloadTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.AnalysisAfterCoreReloadTest: 
[junit4:junit4]    >    1) Thread[id=67, name=qtp661397909-67 Acceptor1 SelectChannelConnector@0.0.0.0:39176, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=68, name=qtp661397909-68 Acceptor2 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=66, name=qtp661397909-66 Acceptor0 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=69, name=qtp661397909-69 Acceptor3 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4502F33E1BD685A]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=67, name=qtp661397909-67 Acceptor1 SelectChannelConnector@0.0.0.0:39176, state=RUNNABLE, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=68, name=qtp661397909-68 Acceptor2 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=66, name=qtp661397909-66 Acceptor0 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=69, name=qtp661397909-69 Acceptor3 SelectChannelConnector@0.0.0.0:39176, state=BLOCKED, group=TGRP-AnalysisAfterCoreReloadTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([D4502F33E1BD685A]:0)
[junit4:junit4] Completed on J0 in 29.41s, 1 test, 2 errors <<< FAILURES!

[...truncated 287 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1159: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 846 ignored

Total time: 18 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 822 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/822/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest:     1) Thread[id=47, name=qtp1984642955-47 Acceptor1 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=97, name=qtp196108812-97 Acceptor2 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=96, name=qtp196108812-96 Acceptor1 SelectChannelConnector@0.0.0.0:53952, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=32, name=qtp2085894369-32 Acceptor3 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=80, name=qtp638812138-80 Acceptor3 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=79, name=qtp638812138-79 Acceptor2 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=78, name=qtp638812138-78 Acceptor1 SelectChannelConnector@0.0.0.0:53946, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=95, name=qtp196108812-95 Acceptor0 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=98, name=qtp196108812-98 Acceptor3 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=48, name=qtp1984642955-48 Acceptor2 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=30, name=qtp2085894369-30 Acceptor1 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=31, name=qtp2085894369-31 Acceptor2 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=77, name=qtp638812138-77 Acceptor0 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=29, name=qtp2085894369-29 Acceptor0 SelectChannelConnector@0.0.0.0:53906, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=46, name=qtp1984642955-46 Acceptor0 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=49, name=qtp1984642955-49 Acceptor3 SelectChannelConnector@0.0.0.0:53922, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 16 threads leaked from SUITE scope at org.apache.solr.cloud.BasicDistributedZkTest: 
   1) Thread[id=47, name=qtp1984642955-47 Acceptor1 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=97, name=qtp196108812-97 Acceptor2 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=96, name=qtp196108812-96 Acceptor1 SelectChannelConnector@0.0.0.0:53952, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=32, name=qtp2085894369-32 Acceptor3 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=80, name=qtp638812138-80 Acceptor3 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=79, name=qtp638812138-79 Acceptor2 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=78, name=qtp638812138-78 Acceptor1 SelectChannelConnector@0.0.0.0:53946, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=95, name=qtp196108812-95 Acceptor0 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=98, name=qtp196108812-98 Acceptor3 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=48, name=qtp1984642955-48 Acceptor2 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=30, name=qtp2085894369-30 Acceptor1 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=31, name=qtp2085894369-31 Acceptor2 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=77, name=qtp638812138-77 Acceptor0 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=29, name=qtp2085894369-29 Acceptor0 SelectChannelConnector@0.0.0.0:53906, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=46, name=qtp1984642955-46 Acceptor0 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=49, name=qtp1984642955-49 Acceptor3 SelectChannelConnector@0.0.0.0:53922, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([4375590A8F5A2C7A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=47, name=qtp1984642955-47 Acceptor1 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=97, name=qtp196108812-97 Acceptor2 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=96, name=qtp196108812-96 Acceptor1 SelectChannelConnector@0.0.0.0:53952, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=32, name=qtp2085894369-32 Acceptor3 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=80, name=qtp638812138-80 Acceptor3 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=78, name=qtp638812138-78 Acceptor1 SelectChannelConnector@0.0.0.0:53946, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=79, name=qtp638812138-79 Acceptor2 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=95, name=qtp196108812-95 Acceptor0 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=48, name=qtp1984642955-48 Acceptor2 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=98, name=qtp196108812-98 Acceptor3 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=30, name=qtp2085894369-30 Acceptor1 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=77, name=qtp638812138-77 Acceptor0 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=31, name=qtp2085894369-31 Acceptor2 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=29, name=qtp2085894369-29 Acceptor0 SelectChannelConnector@0.0.0.0:53906, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=46, name=qtp1984642955-46 Acceptor0 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=49, name=qtp1984642955-49 Acceptor3 SelectChannelConnector@0.0.0.0:53922, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=47, name=qtp1984642955-47 Acceptor1 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=97, name=qtp196108812-97 Acceptor2 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=96, name=qtp196108812-96 Acceptor1 SelectChannelConnector@0.0.0.0:53952, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=32, name=qtp2085894369-32 Acceptor3 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=80, name=qtp638812138-80 Acceptor3 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=78, name=qtp638812138-78 Acceptor1 SelectChannelConnector@0.0.0.0:53946, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=79, name=qtp638812138-79 Acceptor2 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=95, name=qtp196108812-95 Acceptor0 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=48, name=qtp1984642955-48 Acceptor2 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=98, name=qtp196108812-98 Acceptor3 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=30, name=qtp2085894369-30 Acceptor1 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=77, name=qtp638812138-77 Acceptor0 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=31, name=qtp2085894369-31 Acceptor2 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=29, name=qtp2085894369-29 Acceptor0 SelectChannelConnector@0.0.0.0:53906, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=46, name=qtp1984642955-46 Acceptor0 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=49, name=qtp1984642955-49 Acceptor3 SelectChannelConnector@0.0.0.0:53922, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([4375590A8F5A2C7A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
12 threads leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest:     1) Thread[id=91, name=qtp1306894539-91 Acceptor3 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=108, name=qtp1160514367-108 Acceptor2 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=90, name=qtp1306894539-90 Acceptor2 SelectChannelConnector@0.0.0.0:54001, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=89, name=qtp1306894539-89 Acceptor1 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=41, name=qtp70632475-41 Acceptor3 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=38, name=qtp70632475-38 Acceptor0 SelectChannelConnector@0.0.0.0:53943, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=107, name=qtp1160514367-107 Acceptor1 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=109, name=qtp1160514367-109 Acceptor3 SelectChannelConnector@0.0.0.0:54066, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=40, name=qtp70632475-40 Acceptor2 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=106, name=qtp1160514367-106 Acceptor0 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=88, name=qtp1306894539-88 Acceptor0 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=39, name=qtp70632475-39 Acceptor1 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 12 threads leaked from SUITE scope at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest: 
   1) Thread[id=91, name=qtp1306894539-91 Acceptor3 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=108, name=qtp1160514367-108 Acceptor2 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=90, name=qtp1306894539-90 Acceptor2 SelectChannelConnector@0.0.0.0:54001, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=89, name=qtp1306894539-89 Acceptor1 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=41, name=qtp70632475-41 Acceptor3 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=38, name=qtp70632475-38 Acceptor0 SelectChannelConnector@0.0.0.0:53943, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=107, name=qtp1160514367-107 Acceptor1 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=109, name=qtp1160514367-109 Acceptor3 SelectChannelConnector@0.0.0.0:54066, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=40, name=qtp70632475-40 Acceptor2 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=106, name=qtp1160514367-106 Acceptor0 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=88, name=qtp1306894539-88 Acceptor0 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=39, name=qtp70632475-39 Acceptor1 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([4375590A8F5A2C7A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.FullSolrCloudDistribCmdsTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=89, name=qtp1306894539-89 Acceptor1 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=41, name=qtp70632475-41 Acceptor3 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=38, name=qtp70632475-38 Acceptor0 SelectChannelConnector@0.0.0.0:53943, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=91, name=qtp1306894539-91 Acceptor3 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=107, name=qtp1160514367-107 Acceptor1 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=108, name=qtp1160514367-108 Acceptor2 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=90, name=qtp1306894539-90 Acceptor2 SelectChannelConnector@0.0.0.0:54001, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=109, name=qtp1160514367-109 Acceptor3 SelectChannelConnector@0.0.0.0:54066, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=40, name=qtp70632475-40 Acceptor2 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=106, name=qtp1160514367-106 Acceptor0 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=88, name=qtp1306894539-88 Acceptor0 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=39, name=qtp70632475-39 Acceptor1 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=89, name=qtp1306894539-89 Acceptor1 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=41, name=qtp70632475-41 Acceptor3 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=38, name=qtp70632475-38 Acceptor0 SelectChannelConnector@0.0.0.0:53943, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=91, name=qtp1306894539-91 Acceptor3 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=107, name=qtp1160514367-107 Acceptor1 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=108, name=qtp1160514367-108 Acceptor2 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=90, name=qtp1306894539-90 Acceptor2 SelectChannelConnector@0.0.0.0:54001, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=109, name=qtp1160514367-109 Acceptor3 SelectChannelConnector@0.0.0.0:54066, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=40, name=qtp70632475-40 Acceptor2 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=106, name=qtp1160514367-106 Acceptor0 SelectChannelConnector@0.0.0.0:54066, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=88, name=qtp1306894539-88 Acceptor0 SelectChannelConnector@0.0.0.0:54001, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=39, name=qtp70632475-39 Acceptor1 SelectChannelConnector@0.0.0.0:53943, state=BLOCKED, group=TGRP-FullSolrCloudDistribCmdsTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([4375590A8F5A2C7A]:0)




Build Log:
[...truncated 7655 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4:junit4]   2> 4 T22 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-FullSolrCloudDistribCmdsTest-1351617660370
[junit4:junit4]   2> 15 T22 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 107 T23 oaz.Environment.logEnv Server environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT
[junit4:junit4]   2> 107 T23 oaz.Environment.logEnv Server environment:host.name=lucene.zones.apache.org
[junit4:junit4]   2> 107 T23 oaz.Environment.logEnv Server environment:java.version=1.6.0_32
[junit4:junit4]   2> 108 T23 oaz.Environment.logEnv Server environment:java.vendor=Sun Microsystems Inc.
[junit4:junit4]   2> 108 T23 oaz.Environment.logEnv Server environment:java.home=/usr/local/openjdk6/jre
[junit4:junit4]   2> 108 T23 oaz.Environment.logEnv Server environment:java.class.path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-solrj/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/common/lucene-analyzers-common-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/highlighter/lucene-highlighter-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/memory/lucene-memory-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/misc/lucene-misc-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/spatial/lucene-spatial-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/suggest/lucene-suggest-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/grouping/lucene-grouping-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queries/lucene-queries-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queryparser/lucene-queryparser-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-codec-1.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-lang-2.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/guava-r05.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/javax.servlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/spatial4j-0.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/commons-io-2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpcore-4.1.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/log4j-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-jdk14-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/zookeeper-3.3.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-continuation-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-deploy-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-http-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-io-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-jmx-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-security-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-server-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-servlet-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-util-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-webapp-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-xml-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/servlet-api-3.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/randomizedtesting-runner-2.0.4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.2.0.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-log4j.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-logging.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-oro.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-javamail.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bcel.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jdepend.jar:/usr/local/openjdk6/lib/tools.jar:/home/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.0.4.jar
[junit4:junit4]   2> 109 T23 oaz.Environment.logEnv Server environment:java.library.path=/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjdk6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/local/lib
[junit4:junit4]   2> 109 T23 oaz.Environment.logEnv Server environment:java.io.tmpdir=.
[junit4:junit4]   2> 109 T23 oaz.Environment.logEnv Server environment:java.compiler=<NA>
[junit4:junit4]   2> 109 T23 oaz.Environment.logEnv Server environment:os.name=FreeBSD
[junit4:junit4]   2> 110 T23 oaz.Environment.logEnv Server environment:os.arch=amd64
[junit4:junit4]   2> 110 T23 oaz.Environment.logEnv Server environment:os.version=9.0-RELEASE
[junit4:junit4]   2> 110 T23 oaz.Environment.logEnv Server environment:user.name=hudson
[junit4:junit4]   2> 110 T23 oaz.Environment.logEnv Server environment:user.home=/home/hudson
[junit4:junit4]   2> 111 T23 oaz.Environment.logEnv Server environment:user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 151 T23 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 183 T23 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 223 T22 oasc.ZkTestServer.run start zk server on port:53935
[junit4:junit4]   2> 250 T23 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 352 T22 oaz.Environment.logEnv Client environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT
[junit4:junit4]   2> 352 T22 oaz.Environment.logEnv Client environment:host.name=lucene.zones.apache.org
[junit4:junit4]   2> 353 T22 oaz.Environment.logEnv Client environment:java.version=1.6.0_32
[junit4:junit4]   2> 353 T22 oaz.Environment.logEnv Client environment:java.vendor=Sun Microsystems Inc.
[junit4:junit4]   2> 353 T22 oaz.Environment.logEnv Client environment:java.home=/usr/local/openjdk6/jre
[junit4:junit4]   2> 354 T22 oaz.Environment.logEnv Client environment:java.class.path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-solrj/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/common/lucene-analyzers-common-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/highlighter/lucene-highlighter-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/memory/lucene-memory-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/misc/lucene-misc-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/spatial/lucene-spatial-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/suggest/lucene-suggest-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/grouping/lucene-grouping-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queries/lucene-queries-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queryparser/lucene-queryparser-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-codec-1.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-lang-2.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/guava-r05.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/javax.servlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/spatial4j-0.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/commons-io-2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpcore-4.1.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/log4j-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-jdk14-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/zookeeper-3.3.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-continuation-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-deploy-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-http-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-io-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-jmx-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-security-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-server-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-servlet-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-util-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-webapp-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-xml-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/servlet-api-3.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/randomizedtesting-runner-2.0.4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.2.0.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-log4j.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-logging.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-oro.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-javamail.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bcel.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jdepend.jar:/usr/local/openjdk6/lib/tools.jar:/home/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.0.4.jar
[junit4:junit4]   2> 354 T22 oaz.Environment.logEnv Client environment:java.library.path=/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjdk6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/local/lib
[junit4:junit4]   2> 355 T22 oaz.Environment.logEnv Client environment:java.io.tmpdir=.
[junit4:junit4]   2> 355 T22 oaz.Environment.logEnv Client environment:java.compiler=<NA>
[junit4:junit4]   2> 355 T22 oaz.Environment.logEnv Client environment:os.name=FreeBSD
[junit4:junit4]   2> 356 T22 oaz.Environment.logEnv Client environment:os.arch=amd64
[junit4:junit4]   2> 356 T22 oaz.Environment.logEnv Client environment:os.version=9.0-RELEASE
[junit4:junit4]   2> 356 T22 oaz.Environment.logEnv Client environment:user.name=hudson
[junit4:junit4]   2> 357 T22 oaz.Environment.logEnv Client environment:user.home=/home/hudson
[junit4:junit4]   2> 357 T22 oaz.Environment.logEnv Client environment:user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 364 T22 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53935 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@11696cdb
[junit4:junit4]   2> 446 T28 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:53935
[junit4:junit4]   2> 456 T22 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 481 T28 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53935, initiating session
[junit4:junit4]   2> 500 T24 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:53938
[junit4:junit4]   2> 514 T24 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:53938
[junit4:junit4]   2> 526 T26 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 556 T26 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab2b0c7140000 with negotiated timeout 10000 for client /140.211.11.196:53938
[junit4:junit4]   2> 556 T28 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:53935, sessionid = 0x13ab2b0c7140000, negotiated timeout = 10000
[junit4:junit4]   2> 563 T29 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11696cdb name:ZooKeeperConnection Watcher:127.0.0.1:53935 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 563 T22 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 568 T22 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 632 T27 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ab2b0c7140000
[junit4:junit4]   2> 634 T29 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 634 T22 oaz.ZooKeeper.close Session: 0x13ab2b0c7140000 closed
[junit4:junit4]   2> 635 T24 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:53938 which had sessionid 0x13ab2b0c7140000
[junit4:junit4]   2> 635 T22 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53935/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@44d79c75
[junit4:junit4]   2> 636 T30 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:53935
[junit4:junit4]   2> 636 T22 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 638 T30 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53935, initiating session
[junit4:junit4]   2> 638 T24 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:53939
[junit4:junit4]   2> 638 T24 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:53939
[junit4:junit4]   2> 640 T26 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab2b0c7140001 with negotiated timeout 10000 for client /140.211.11.196:53939
[junit4:junit4]   2> 640 T30 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:53935, sessionid = 0x13ab2b0c7140001, negotiated timeout = 10000
[junit4:junit4]   2> 640 T31 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44d79c75 name:ZooKeeperConnection Watcher:127.0.0.1:53935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 641 T22 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 658 T22 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 665 T22 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 669 T22 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 677 T22 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 682 T22 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 703 T22 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 711 T22 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 714 T22 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 821 T22 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 823 T22 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 841 T22 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 842 T22 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 849 T22 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 850 T22 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 854 T22 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 855 T22 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 860 T22 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 860 T22 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 866 T22 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 867 T22 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 872 T22 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 872 T22 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 877 T22 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 877 T22 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 889 T27 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ab2b0c7140001
[junit4:junit4]   2> 890 T31 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 890 T24 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:53939 which had sessionid 0x13ab2b0c7140001
[junit4:junit4]   2> 890 T22 oaz.ZooKeeper.close Session: 0x13ab2b0c7140001 closed
[junit4:junit4]   2> 1354 T22 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1687 T22 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:53943
[junit4:junit4]   2> 1713 T22 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1713 T22 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1714 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1742 T22 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1743 T22 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1743 T22 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1744 T22 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1744 T22 oasc.CoreContainer.<init> New CoreContainer 1908246931
[junit4:junit4]   2> 1745 T22 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1746 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1781 T22 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1795 T22 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:53935/solr
[junit4:junit4]   2> 1815 T22 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53935/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@447bd86d
[junit4:junit4]   2> 1816 T42 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:53935
[junit4:junit4]   2> 1816 T22 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1818 T42 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53935, initiating session
[junit4:junit4]   2> 1818 T24 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:53945
[junit4:junit4]   2> 1819 T24 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:53945
[junit4:junit4]   2> 1822 T26 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab2b0c7140002 with negotiated timeout 5000 for client /140.211.11.196:53945
[junit4:junit4]   2> 1822 T42 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:53935, sessionid = 0x13ab2b0c7140002, negotiated timeout = 5000
[junit4:junit4]   2> 1822 T43 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@447bd86d name:ZooKeeperConnection Watcher:127.0.0.1:53935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1823 T22 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1836 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1843 T22 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1846 T22 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53943_solr
[junit4:junit4]   2> 1850 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:delete cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:53943_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:53943_solr
[junit4:junit4]   2> 1851 T22 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53943_solr
[junit4:junit4]   2> 1855 T22 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1855 T22 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1855 T22 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1856 T22 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1856 T22 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1856 T22 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1857 T22 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1857 T22 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1858 T22 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1858 T22 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1874 T22 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1902 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:delete cxid:0x15 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1904 T22 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1917 T22 oasc.Overseer.start Overseer (id=88579615011110914-127.0.0.1:53943_solr-n_0000000000) starting
[junit4:junit4]   2> 1918 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1931 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1932 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:create cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1938 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1939 T45 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1940 T22 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1948 T22 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1970 T44 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 2003 T22 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2004 T22 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 2005 T22 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 2005 T22 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 2007 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2008 T22 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2009 T22 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2157 T22 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2249 T22 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2350 T22 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2373 T22 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2958 T22 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2972 T22 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2975 T22 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 3002 T22 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3008 T22 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3012 T22 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3016 T22 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3020 T22 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/control/data/
[junit4:junit4]   2> 3024 T22 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e37d490
[junit4:junit4]   2> 3059 T22 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/control/data/index/
[junit4:junit4]   2> 3060 T22 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 3061 T22 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/control/data/index forceNew:false
[junit4:junit4]   2> 3069 T22 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c571cc4; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3069 T22 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3070 T22 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 3073 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 3073 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 3074 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 3074 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 3075 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 3076 T22 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 3077 T22 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 3078 T22 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 3079 T22 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 3080 T22 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 3080 T22 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 3081 T22 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 3082 T22 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 3082 T22 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 3083 T22 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 3084 T22 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 3084 T22 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3085 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3086 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3087 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3088 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3089 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3090 T22 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 3090 T22 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 3091 T22 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 3098 T22 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 3102 T22 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 3104 T22 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 3108 T22 oass.SolrIndexSearcher.<init> Opening Searcher@4f57011e main
[junit4:junit4]   2> 3129 T22 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 3130 T22 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 3131 T22 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 3138 T22 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 3214 T50 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4f57011e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 3221 T22 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3222 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:create cxid:0x56 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 3234 T22 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 3235 T22 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:53943/solr shardId:control_shard
[junit4:junit4]   2> 3248 T22 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 3263 T22 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/control_shard/election
[junit4:junit4]   2> 3280 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:delete cxid:0x65 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 3288 T22 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 3290 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:create cxid:0x66 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3480 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3482 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53943_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53943/solr"}
[junit4:junit4]   2> 3497 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:create cxid:0x6d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 3537 T43 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3794 T22 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 3795 T22 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 3795 T22 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53943/solr/collection1/
[junit4:junit4]   2> 3796 T22 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 3797 T22 oasc.SyncStrategy.syncToMe http://127.0.0.1:53943/solr/collection1/ has no replicas
[junit4:junit4]   2> 3797 T22 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53943/solr/collection1/
[junit4:junit4]   2> 3798 T22 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/control_shard
[junit4:junit4]   2> 3808 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140002 type:create cxid:0x8b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4046 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4066 T43 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 4076 T22 oasc.ZkController.register We are http://127.0.0.1:53943/solr/collection1/ and leader is http://127.0.0.1:53943/solr/collection1/
[junit4:junit4]   2> 4082 T22 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53943/solr
[junit4:junit4]   2> 4083 T22 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 4083 T22 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 4087 T22 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4088 T22 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 4089 T22 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 4205 T22 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4452 T22 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 4456 T22 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:53953
[junit4:junit4]   2> 4458 T22 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4459 T22 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351617664599
[junit4:junit4]   2> 4459 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351617664599/'
[junit4:junit4]   2> 4509 T22 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4509 T22 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4510 T22 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351617664599
[junit4:junit4]   2> 4511 T22 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351617664599/solr.xml
[junit4:junit4]   2> 4511 T22 oasc.CoreContainer.<init> New CoreContainer 232003191
[junit4:junit4]   2> 4512 T22 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351617664599/'
[junit4:junit4]   2> 4513 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351617664599/'
[junit4:junit4]   2> 4542 T22 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4561 T22 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:53935/solr
[junit4:junit4]   2> 4562 T22 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53935/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@4b0bc3c9
[junit4:junit4]   2> 4563 T60 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:53935
[junit4:junit4]   2> 4563 T22 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4565 T60 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53935, initiating session
[junit4:junit4]   2> 4565 T24 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:53954
[junit4:junit4]   2> 4566 T24 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:53954
[junit4:junit4]   2> 4569 T26 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab2b0c7140003 with negotiated timeout 5000 for client /140.211.11.196:53954
[junit4:junit4]   2> 4569 T60 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:53935, sessionid = 0x13ab2b0c7140003, negotiated timeout = 5000
[junit4:junit4]   2> 4569 T61 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b0bc3c9 name:ZooKeeperConnection Watcher:127.0.0.1:53935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4570 T22 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4571 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140003 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4573 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4576 T22 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53953_solr
[junit4:junit4]   2> 4576 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4580 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140003 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:53953_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:53953_solr
[junit4:junit4]   2> 4581 T22 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53953_solr
[junit4:junit4]   2> 4582 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53943_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53943/solr"}
[junit4:junit4]   2> 4595 T22 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 4596 T22 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4596 T22 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 4597 T22 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4597 T22 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4598 T22 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4598 T22 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4599 T22 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4600 T22 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4600 T22 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 4607 T43 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4608 T43 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 4629 T22 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 4639 T22 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351617664599/collection1
[junit4:junit4]   2> 4639 T22 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 4640 T22 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 4641 T22 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 4643 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351617664599/collection1/'
[junit4:junit4]   2> 4645 T22 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty1-1351617664599/collection1/lib/README' to classloader
[junit4:junit4]   2> 4646 T22 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty1-1351617664599/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 4716 T22 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 4805 T22 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 4906 T22 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 4926 T22 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 5667 T22 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5681 T22 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5684 T22 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5707 T22 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5713 T22 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5718 T22 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5721 T22 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5723 T22 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351617664599/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty1/
[junit4:junit4]   2> 5724 T22 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e37d490
[junit4:junit4]   2> 5725 T22 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty1/index/
[junit4:junit4]   2> 5726 T22 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5727 T22 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty1/index forceNew:false
[junit4:junit4]   2> 5736 T22 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@a0ccc96; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5736 T22 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5737 T22 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 5740 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5740 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 5741 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 5742 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 5742 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 5743 T22 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 5743 T22 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 5744 T22 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5745 T22 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5746 T22 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 5747 T22 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 5748 T22 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5749 T22 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 5749 T22 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 5750 T22 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 5751 T22 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5752 T22 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5753 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5754 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5755 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5756 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5757 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5759 T22 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 5760 T22 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 5761 T22 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 5769 T22 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5773 T22 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 5774 T22 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 5778 T22 oass.SolrIndexSearcher.<init> Opening Searcher@759f6a57 main
[junit4:junit4]   2> 5779 T22 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5779 T22 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5780 T22 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 5795 T22 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 5849 T66 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@759f6a57 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5855 T22 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6115 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6116 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53953_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53953/solr"}
[junit4:junit4]   2> 6116 T44 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 6136 T61 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6136 T43 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6358 T22 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 6359 T22 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:53953/solr shardId:shard1
[junit4:junit4]   2> 6360 T22 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 6379 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140003 type:delete cxid:0x42 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 6380 T22 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 6381 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140003 type:create cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6391 T22 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 6392 T22 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 6392 T22 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53953/solr/collection1/
[junit4:junit4]   2> 6393 T22 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 6394 T22 oasc.SyncStrategy.syncToMe http://127.0.0.1:53953/solr/collection1/ has no replicas
[junit4:junit4]   2> 6394 T22 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53953/solr/collection1/
[junit4:junit4]   2> 6395 T22 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 6405 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140003 type:create cxid:0x4d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6642 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6663 T43 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6663 T61 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6714 T22 oasc.ZkController.register We are http://127.0.0.1:53953/solr/collection1/ and leader is http://127.0.0.1:53953/solr/collection1/
[junit4:junit4]   2> 6714 T22 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53953/solr
[junit4:junit4]   2> 6715 T22 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 6715 T22 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6718 T22 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6720 T22 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 6720 T22 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 6721 T22 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 6902 T22 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 6906 T22 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:53988
[junit4:junit4]   2> 6907 T22 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6907 T22 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351617667103
[junit4:junit4]   2> 6908 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351617667103/'
[junit4:junit4]   2> 6955 T22 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 6955 T22 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6956 T22 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351617667103
[junit4:junit4]   2> 6957 T22 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351617667103/solr.xml
[junit4:junit4]   2> 6957 T22 oasc.CoreContainer.<init> New CoreContainer 1739173940
[junit4:junit4]   2> 6958 T22 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351617667103/'
[junit4:junit4]   2> 6958 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351617667103/'
[junit4:junit4]   2> 6986 T22 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 7005 T22 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:53935/solr
[junit4:junit4]   2> 7005 T22 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53935/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@24b950d1
[junit4:junit4]   2> 7007 T76 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:53935
[junit4:junit4]   2> 7007 T22 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7008 T76 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53935, initiating session
[junit4:junit4]   2> 7008 T24 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:53989
[junit4:junit4]   2> 7009 T24 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:53989
[junit4:junit4]   2> 7012 T26 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab2b0c7140004 with negotiated timeout 5000 for client /140.211.11.196:53989
[junit4:junit4]   2> 7012 T76 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:53935, sessionid = 0x13ab2b0c7140004, negotiated timeout = 5000
[junit4:junit4]   2> 7012 T77 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24b950d1 name:ZooKeeperConnection Watcher:127.0.0.1:53935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7013 T22 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7014 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140004 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7016 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140004 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7018 T22 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53988_solr
[junit4:junit4]   2> 7019 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140004 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:53988_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:53988_solr
[junit4:junit4]   2> 7026 T22 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53988_solr
[junit4:junit4]   2> 7030 T61 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 7030 T43 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 7031 T22 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 7032 T22 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 7032 T22 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 7033 T22 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 7033 T22 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 7034 T22 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 7035 T22 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 7035 T22 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 7036 T22 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 7036 T22 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 7058 T22 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 7068 T22 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351617667103/collection1
[junit4:junit4]   2> 7069 T22 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 7070 T22 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 7071 T22 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 7073 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351617667103/collection1/'
[junit4:junit4]   2> 7075 T22 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty2-1351617667103/collection1/lib/README' to classloader
[junit4:junit4]   2> 7076 T22 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty2-1351617667103/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 7138 T22 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 7178 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7182 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53953_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53953/solr"}
[junit4:junit4]   2> 7195 T61 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7195 T43 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7195 T77 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7216 T22 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 7318 T22 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 7338 T22 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 8109 T22 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 8153 T22 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 8170 T22 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 8202 T22 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 8209 T22 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 8216 T22 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 8220 T22 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 8224 T22 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351617667103/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty2/
[junit4:junit4]   2> 8225 T22 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e37d490
[junit4:junit4]   2> 8227 T22 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty2/index/
[junit4:junit4]   2> 8228 T22 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 8229 T22 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty2/index forceNew:false
[junit4:junit4]   2> 8271 T22 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@6950ecc7; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 8272 T22 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 8273 T22 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 8277 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 8278 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 8279 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 8279 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 8281 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 8282 T22 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 8282 T22 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 8284 T22 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 8285 T22 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 8286 T22 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 8287 T22 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 8288 T22 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 8289 T22 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 8289 T22 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 8290 T22 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 8291 T22 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 8292 T22 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8293 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8294 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8295 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8296 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8297 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8298 T22 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 8299 T22 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 8300 T22 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 8308 T22 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 8313 T22 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 8315 T22 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 8320 T22 oass.SolrIndexSearcher.<init> Opening Searcher@45e6612c main
[junit4:junit4]   2> 8322 T22 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 8323 T22 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 8324 T22 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 8383 T22 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 8463 T82 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@45e6612c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 8472 T22 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8708 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8710 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53988_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53988/solr"}
[junit4:junit4]   2> 8726 T43 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8726 T77 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8726 T61 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8976 T22 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 8976 T22 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:53988/solr shardId:shard2
[junit4:junit4]   2> 8978 T22 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 9001 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140004 type:delete cxid:0x42 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 9003 T22 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 9004 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140004 type:create cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 9014 T22 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 9014 T22 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 9015 T22 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53988/solr/collection1/
[junit4:junit4]   2> 9016 T22 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 9017 T22 oasc.SyncStrategy.syncToMe http://127.0.0.1:53988/solr/collection1/ has no replicas
[junit4:junit4]   2> 9018 T22 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53988/solr/collection1/
[junit4:junit4]   2> 9019 T22 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 9024 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140004 type:create cxid:0x4c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 9232 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9249 T43 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 9250 T61 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 9250 T77 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 9290 T22 oasc.ZkController.register We are http://127.0.0.1:53988/solr/collection1/ and leader is http://127.0.0.1:53988/solr/collection1/
[junit4:junit4]   2> 9290 T22 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53988/solr
[junit4:junit4]   2> 9291 T22 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 9291 T22 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 9294 T22 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9295 T22 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 9296 T22 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 9296 T22 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 9408 T22 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 9411 T22 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:54001
[junit4:junit4]   2> 9412 T22 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 9413 T22 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351617669671
[junit4:junit4]   2> 9413 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351617669671/'
[junit4:junit4]   2> 9439 T22 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 9440 T22 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 9440 T22 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351617669671
[junit4:junit4]   2> 9441 T22 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351617669671/solr.xml
[junit4:junit4]   2> 9441 T22 oasc.CoreContainer.<init> New CoreContainer 2098023134
[junit4:junit4]   2> 9442 T22 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351617669671/'
[junit4:junit4]   2> 9442 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351617669671/'
[junit4:junit4]   2> 9461 T22 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 9472 T22 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:53935/solr
[junit4:junit4]   2> 9473 T22 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:53935/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@4280bc28
[junit4:junit4]   2> 9474 T92 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:53935
[junit4:junit4]   2> 9474 T22 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 9475 T92 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:53935, initiating session
[junit4:junit4]   2> 9476 T24 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:54003
[junit4:junit4]   2> 9476 T24 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:54003
[junit4:junit4]   2> 9479 T26 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab2b0c7140005 with negotiated timeout 5000 for client /140.211.11.196:54003
[junit4:junit4]   2> 9479 T92 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:53935, sessionid = 0x13ab2b0c7140005, negotiated timeout = 5000
[junit4:junit4]   2> 9479 T93 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4280bc28 name:ZooKeeperConnection Watcher:127.0.0.1:53935/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 9480 T22 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 9480 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140005 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 9482 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140005 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 9496 T22 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54001_solr
[junit4:junit4]   2> 9497 T27 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab2b0c7140005 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:54001_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:54001_solr
[junit4:junit4]   2> 9498 T22 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54001_solr
[junit4:junit4]   2> 9501 T43 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 9502 T61 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 9502 T77 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 9502 T22 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 9503 T22 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 9503 T22 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 9503 T22 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 9504 T22 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 9504 T22 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 9505 T22 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 9505 T22 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 9505 T22 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 9506 T22 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 9522 T22 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9528 T22 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351617669671/collection1
[junit4:junit4]   2> 9529 T22 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9530 T22 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9530 T22 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9532 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351617669671/collection1/'
[junit4:junit4]   2> 9533 T22 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty3-1351617669671/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9533 T22 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty3-1351617669671/collection1/lib/README' to classloader
[junit4:junit4]   2> 9574 T22 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 9629 T22 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 9730 T22 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 9742 T22 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 9758 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9759 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:53988_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:53988/solr"}
[junit4:junit4]   2> 9779 T77 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9779 T61 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9779 T93 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 9779 T43 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10215 T22 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10231 T22 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10236 T22 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10268 T22 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10275 T22 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10281 T22 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10284 T22 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10287 T22 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty3-1351617669671/collection1/, dataDir=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty3/
[junit4:junit4]   2> 10288 T22 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6e37d490
[junit4:junit4]   2> 10289 T22 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty3/index/
[junit4:junit4]   2> 10290 T22 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10292 T22 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty3/index forceNew:false
[junit4:junit4]   2> 10300 T22 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.FullSolrCloudDistribCmdsTest-1351617660370/jetty3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1599f755; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10300 T22 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 10302 T22 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 10305 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10306 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 10307 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10308 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 10309 T22 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10309 T22 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10310 T22 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10311 T22 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10312 T22 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10313 T22 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 10314 T22 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 10315 T22 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10316 T22 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 10317 T22 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 10318 T22 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 10319 T22 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10320 T22 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10321 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10322 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10323 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10324 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10325 T22 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10326 T22 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 10326 T22 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 10327 T22 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 10335 T22 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10340 T22 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 10343 T22 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 10347 T22 oass.SolrIndexSearcher.<init> Opening Searcher@4c5e43ee main
[junit4:junit4]   2> 10349 T22 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10350 T22 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10351 T22 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 10359 T22 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 10415 T98 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c5e43ee main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10430 T22 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 10787 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10789 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"2",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:54001_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:54001/solr"}
[junit4:junit4]   2> 10804 T77 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10804 T61 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10804 T43 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10804 T93 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (4)
[junit4:junit4]   2> 10934 T22 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 10934 T22 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:54001/solr shardId:shard1
[junit4:junit4]   2> 10939 T22 oasc.ZkController.register We are http://127.0.0.1:54001/solr/collection1/ and leader is http://127.0.0.1:53953/solr/collection1/
[junit4:junit4]   2> 10939 T22 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54001/solr
[junit4:junit4]   2> 10940 T22 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 10940 T22 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@6dcbf6bb url=http://127.0.0.1:54001/solr/collection1 node=127.0.0.1:54001_solr
[junit4:junit4]   2> 10948 T99 C4 P54001 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 10948 T22 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10949 T99 C4 P54001 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 10949 T99 C4 P54001 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 10950 T22 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 10951 T22 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 10951 T99 C4 P54001 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53953/solr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 10951 T22 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 10961 T99 C4 P54001 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 10985 T99 C4 P54001 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:54001/solr START replicas=[http://127.0.0.1:53953/solr/collection1/] nUpdates=100
[junit4:junit4]   2> 11007 T99 C4 P54001 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 11008 T99 C4 P54001 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 11008 T99 C4 P54001 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 11009 T99 C4 P54001 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 11163 T22 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 11169 T22 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:54066
[junit4:junit4]   2> 11170 T22 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 11171 T22 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351617671334
[junit4:junit4]   2> 11171 T22 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty4-1351617671334/'
[junit4:junit4]   2> 11310 T44 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11312 T44 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]

[...truncated too long message...]

ver.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=79, name=qtp638812138-79 Acceptor2 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=95, name=qtp196108812-95 Acceptor0 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    9) Thread[id=48, name=qtp1984642955-48 Acceptor2 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   10) Thread[id=98, name=qtp196108812-98 Acceptor3 SelectChannelConnector@0.0.0.0:53952, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=30, name=qtp2085894369-30 Acceptor1 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=77, name=qtp638812138-77 Acceptor0 SelectChannelConnector@0.0.0.0:53946, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=31, name=qtp2085894369-31 Acceptor2 SelectChannelConnector@0.0.0.0:53906, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=29, name=qtp2085894369-29 Acceptor0 SelectChannelConnector@0.0.0.0:53906, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=46, name=qtp1984642955-46 Acceptor0 SelectChannelConnector@0.0.0.0:53922, state=BLOCKED, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=49, name=qtp1984642955-49 Acceptor3 SelectChannelConnector@0.0.0.0:53922, state=RUNNABLE, group=TGRP-BasicDistributedZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([4375590A8F5A2C7A]:0)
[junit4:junit4] Completed on J0 in 334.08s, 1 test, 2 errors <<< FAILURES!

[...truncated 179 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1159: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 848 ignored

Total time: 22 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 821 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/821/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
8 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest:     1) Thread[id=79, name=qtp64462389-79 Acceptor1 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=81, name=qtp64462389-81 Acceptor3 SelectChannelConnector@0.0.0.0:61401, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=80, name=qtp64462389-80 Acceptor2 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=78, name=qtp64462389-78 Acceptor0 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=169, name=qtp411693457-169 Acceptor2 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=167, name=qtp411693457-167 Acceptor0 SelectChannelConnector@0.0.0.0:13544, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=168, name=qtp411693457-168 Acceptor1 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=170, name=qtp411693457-170 Acceptor3 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 8 threads leaked from SUITE scope at org.apache.solr.cloud.RecoveryZkTest: 
   1) Thread[id=79, name=qtp64462389-79 Acceptor1 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=81, name=qtp64462389-81 Acceptor3 SelectChannelConnector@0.0.0.0:61401, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=80, name=qtp64462389-80 Acceptor2 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=78, name=qtp64462389-78 Acceptor0 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=169, name=qtp411693457-169 Acceptor2 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=167, name=qtp411693457-167 Acceptor0 SelectChannelConnector@0.0.0.0:13544, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=168, name=qtp411693457-168 Acceptor1 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=170, name=qtp411693457-170 Acceptor3 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([20E679A9BD29B952]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=79, name=qtp64462389-79 Acceptor1 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=81, name=qtp64462389-81 Acceptor3 SelectChannelConnector@0.0.0.0:61401, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=80, name=qtp64462389-80 Acceptor2 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=78, name=qtp64462389-78 Acceptor0 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=169, name=qtp411693457-169 Acceptor2 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=167, name=qtp411693457-167 Acceptor0 SelectChannelConnector@0.0.0.0:13544, state=RUNNABLE, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=168, name=qtp411693457-168 Acceptor1 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=170, name=qtp411693457-170 Acceptor3 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=79, name=qtp64462389-79 Acceptor1 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=81, name=qtp64462389-81 Acceptor3 SelectChannelConnector@0.0.0.0:61401, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=80, name=qtp64462389-80 Acceptor2 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=78, name=qtp64462389-78 Acceptor0 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=169, name=qtp411693457-169 Acceptor2 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=167, name=qtp411693457-167 Acceptor0 SelectChannelConnector@0.0.0.0:13544, state=RUNNABLE, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=168, name=qtp411693457-168 Acceptor1 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=170, name=qtp411693457-170 Acceptor3 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([20E679A9BD29B952]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.NoCacheHeaderTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest:     1) Thread[id=15, name=qtp1668782213-15 Acceptor3 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=20, name=qtp1668782213-20 Acceptor1 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=22, name=qtp1668782213-22 Acceptor2 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=21, name=qtp1668782213-21 Acceptor0 SelectChannelConnector@0.0.0.0:14366, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest: 
   1) Thread[id=15, name=qtp1668782213-15 Acceptor3 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=20, name=qtp1668782213-20 Acceptor1 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=22, name=qtp1668782213-22 Acceptor2 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=21, name=qtp1668782213-21 Acceptor0 SelectChannelConnector@0.0.0.0:14366, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([20E679A9BD29B952]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.NoCacheHeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=15, name=qtp1668782213-15 Acceptor3 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=20, name=qtp1668782213-20 Acceptor1 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=22, name=qtp1668782213-22 Acceptor2 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=21, name=qtp1668782213-21 Acceptor0 SelectChannelConnector@0.0.0.0:14366, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=15, name=qtp1668782213-15 Acceptor3 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=20, name=qtp1668782213-20 Acceptor1 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=22, name=qtp1668782213-22 Acceptor2 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=21, name=qtp1668782213-21 Acceptor0 SelectChannelConnector@0.0.0.0:14366, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([20E679A9BD29B952]:0)




Build Log:
[...truncated 7658 lines...]
[junit4:junit4] Suite: org.apache.solr.servlet.NoCacheHeaderTest
[junit4:junit4]   2> 190 T12 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./solrtest-NoCacheHeaderTest-1351595860535
[junit4:junit4]   2> 195 T12 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 817 T12 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1189 T12 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:14366
[junit4:junit4]   2> 1395 T12 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1396 T12 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1396 T12 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1960 T12 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1963 T12 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1964 T12 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1992 T12 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1996 T12 oasc.CoreContainer.<init> New CoreContainer 1833717089
[junit4:junit4]   2> 1997 T12 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1998 T12 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2062 T12 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2134 T12 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2135 T12 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2136 T12 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2137 T12 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2201 T12 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2331 T12 oasc.SolrConfig.<init> Loaded SolrConfig: solr/collection1/conf/solrconfig-nocache.xml
[junit4:junit4]   2> 2360 T12 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2418 T12 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 4614 T12 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 4646 T12 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 4650 T12 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5057 T12 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5072 T12 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5077 T12 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5096 T12 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5204 T12 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351595860535/
[junit4:junit4]   2> 5205 T12 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 5262 T12 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351595860535/index/
[junit4:junit4]   2> 5263 T12 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351595860535/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5344 T12 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/solrtest-NoCacheHeaderTest-1351595860535/index forceNew:false
[junit4:junit4]   2> 5746 T12 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@6a2d0483 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f549ceb),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5747 T12 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 6750 T12 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6823 T12 oass.SolrIndexSearcher.<init> Opening Searcher@1f5fa713 main
[junit4:junit4]   2> 6824 T12 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 6905 T12 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6906 T12 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7032 T12 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 7033 T12 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 7033 T12 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 7034 T12 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 7034 T12 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 7035 T12 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 7035 T12 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 7036 T12 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 7036 T12 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 7078 T12 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 7554 T12 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 7554 T27 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f5fa713 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 7555 T12 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 7556 T12 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 7557 T12 oas.SolrJettyTestBase.createJetty Jetty Assigned Port#14366
[junit4:junit4]   2> 7565 T12 oas.SolrTestCaseJ4.setUp ###Starting testLastModified
[junit4:junit4]   2> 7594 T12 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@58b45130
[junit4:junit4]   2> 9421 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=514 
[junit4:junit4]   2> 9662 T24 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9675 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9680 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9684 T24 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9688 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9692 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9696 T24 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9700 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9704 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9705 T12 oas.SolrTestCaseJ4.tearDown ###Ending testLastModified
[junit4:junit4]   2> 9722 T12 oas.SolrTestCaseJ4.setUp ###Starting testEtag
[junit4:junit4]   2> 9726 T24 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9730 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9734 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9738 T24 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9742 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9746 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9750 T24 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9754 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9758 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9762 T24 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9763 T12 oas.SolrTestCaseJ4.tearDown ###Ending testEtag
[junit4:junit4]   2> 9779 T12 oas.SolrTestCaseJ4.setUp ###Starting testCacheControl
[junit4:junit4]   2> 9783 T26 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9787 T23 C0 REQ [collection1] webapp=/solr path=/select params={qt=standard&q=solr} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 9796 T24 C0 REQ [collection1] webapp=/solr path=/select params={q=solr&qt=standard} hits=0 status=0 QTime=0 
[junit4:junit4]   2> 9797 T12 oas.SolrTestCaseJ4.tearDown ###Ending testCacheControl
[junit4:junit4]   2> 9847 T12 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1833717089
[junit4:junit4]   2> 9848 T12 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@58b45130
[junit4:junit4]   2> 9848 T12 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 9849 T12 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 9850 T12 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 9851 T12 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 9856 T12 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 9980 T12 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 10082 T12 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 10082 T12 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 10114 T11 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 30232 T11 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest: 
[junit4:junit4]   2> 	   1) Thread[id=15, name=qtp1668782213-15 Acceptor3 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=20, name=qtp1668782213-20 Acceptor1 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=22, name=qtp1668782213-22 Acceptor2 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=21, name=qtp1668782213-21 Acceptor0 SelectChannelConnector@0.0.0.0:14366, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 30233 T11 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=15, name=qtp1668782213-15 Acceptor3 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	   2) Thread[id=20, name=qtp1668782213-20 Acceptor1 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	   3) Thread[id=22, name=qtp1668782213-22 Acceptor2 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	   4) Thread[id=21, name=qtp1668782213-21 Acceptor0 SelectChannelConnector@0.0.0.0:14366, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 33277 T11 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=15, name=qtp1668782213-15 Acceptor3 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=20, name=qtp1668782213-20 Acceptor1 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=22, name=qtp1668782213-22 Acceptor2 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=21, name=qtp1668782213-21 Acceptor0 SelectChannelConnector@0.0.0.0:14366, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=en_SG, timezone=Brazil/East
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=73647552,total=253886464
[junit4:junit4]   2> NOTE: All tests run in this JVM: [SystemInfoHandlerTest, NoCacheHeaderTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=NoCacheHeaderTest -Dtests.seed=20E679A9BD29B952 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_SG -Dtests.timezone=Brazil/East -Dtests.file.encoding=US-ASCII
[junit4:junit4] ERROR   0.00s J0 | NoCacheHeaderTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.NoCacheHeaderTest: 
[junit4:junit4]    >    1) Thread[id=15, name=qtp1668782213-15 Acceptor3 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=20, name=qtp1668782213-20 Acceptor1 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=22, name=qtp1668782213-22 Acceptor2 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=21, name=qtp1668782213-21 Acceptor0 SelectChannelConnector@0.0.0.0:14366, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([20E679A9BD29B952]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=15, name=qtp1668782213-15 Acceptor3 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=20, name=qtp1668782213-20 Acceptor1 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=22, name=qtp1668782213-22 Acceptor2 SelectChannelConnector@0.0.0.0:14366, state=BLOCKED, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=21, name=qtp1668782213-21 Acceptor0 SelectChannelConnector@0.0.0.0:14366, state=RUNNABLE, group=TGRP-NoCacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([20E679A9BD29B952]:0)
[junit4:junit4] Completed on J0 in 33.77s, 3 tests, 2 errors <<< FAILURES!

[...truncated 514 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4]   2> 3 T62 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-RecoveryZkTest-1351595889579
[junit4:junit4]   2> 9 T62 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 124 T63 oaz.Environment.logEnv Server environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT
[junit4:junit4]   2> 125 T63 oaz.Environment.logEnv Server environment:host.name=lucene.zones.apache.org
[junit4:junit4]   2> 125 T63 oaz.Environment.logEnv Server environment:java.version=1.6.0_32
[junit4:junit4]   2> 125 T63 oaz.Environment.logEnv Server environment:java.vendor=Sun Microsystems Inc.
[junit4:junit4]   2> 125 T63 oaz.Environment.logEnv Server environment:java.home=/usr/local/openjdk6/jre
[junit4:junit4]   2> 126 T63 oaz.Environment.logEnv Server environment:java.class.path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-solrj/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/common/lucene-analyzers-common-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/highlighter/lucene-highlighter-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/memory/lucene-memory-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/misc/lucene-misc-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/spatial/lucene-spatial-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/suggest/lucene-suggest-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/grouping/lucene-grouping-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queries/lucene-queries-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queryparser/lucene-queryparser-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-codec-1.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-lang-2.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/guava-r05.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/javax.servlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/spatial4j-0.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/commons-io-2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpcore-4.1.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/log4j-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-jdk14-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/zookeeper-3.3.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-continuation-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-deploy-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-http-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-io-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-jmx-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-security-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-server-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-servlet-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-util-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-webapp-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-xml-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/servlet-api-3.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/randomizedtesting-runner-2.0.4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.2.0.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-log4j.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-logging.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-oro.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-javamail.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bcel.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jdepend.jar:/usr/local/openjdk6/lib/tools.jar:/home/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.0.4.jar
[junit4:junit4]   2> 127 T63 oaz.Environment.logEnv Server environment:java.library.path=/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjdk6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/local/lib
[junit4:junit4]   2> 127 T63 oaz.Environment.logEnv Server environment:java.io.tmpdir=.
[junit4:junit4]   2> 127 T63 oaz.Environment.logEnv Server environment:java.compiler=<NA>
[junit4:junit4]   2> 128 T63 oaz.Environment.logEnv Server environment:os.name=FreeBSD
[junit4:junit4]   2> 128 T63 oaz.Environment.logEnv Server environment:os.arch=amd64
[junit4:junit4]   2> 128 T63 oaz.Environment.logEnv Server environment:os.version=9.0-RELEASE
[junit4:junit4]   2> 128 T63 oaz.Environment.logEnv Server environment:user.name=hudson
[junit4:junit4]   2> 129 T63 oaz.Environment.logEnv Server environment:user.home=/home/hudson
[junit4:junit4]   2> 129 T63 oaz.Environment.logEnv Server environment:user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 179 T63 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 219 T63 oazs.NIOServerCnxn$Factory.<init> binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 295 T63 oazsp.FileTxnSnapLog.save Snapshotting: 0
[junit4:junit4]   2> 317 T62 oasc.ZkTestServer.run start zk server on port:63131
[junit4:junit4]   2> 432 T62 oaz.Environment.logEnv Client environment:zookeeper.version=3.3.6-1366786, built on 07/29/2012 06:22 GMT
[junit4:junit4]   2> 433 T62 oaz.Environment.logEnv Client environment:host.name=lucene.zones.apache.org
[junit4:junit4]   2> 433 T62 oaz.Environment.logEnv Client environment:java.version=1.6.0_32
[junit4:junit4]   2> 434 T62 oaz.Environment.logEnv Client environment:java.vendor=Sun Microsystems Inc.
[junit4:junit4]   2> 434 T62 oaz.Environment.logEnv Client environment:java.home=/usr/local/openjdk6/jre
[junit4:junit4]   2> 434 T62 oaz.Environment.logEnv Client environment:java.class.path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/test:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/test-framework/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/codecs/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-solrj/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/common/lucene-analyzers-common-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/highlighter/lucene-highlighter-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/memory/lucene-memory-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/misc/lucene-misc-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/spatial/lucene-spatial-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/suggest/lucene-suggest-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/grouping/lucene-grouping-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queries/lucene-queries-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/queryparser/lucene-queryparser-4.1-SNAPSHOT.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-cli-1.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-codec-1.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-fileupload-1.2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/commons-lang-2.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/easymock-2.2.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/guava-r05.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/javax.servlet-api-3.0.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/lib/spatial4j-0.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/commons-io-2.1.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpclient-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpcore-4.1.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/httpmime-4.1.3.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/jcl-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/log4j-over-slf4j-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-api-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/slf4j-jdk14-1.6.4.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/wstx-asl-3.2.7.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/solrj/lib/zookeeper-3.3.6.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-continuation-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-deploy-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-http-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-io-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-jmx-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-security-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-server-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-servlet-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-util-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-webapp-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/jetty-xml-8.1.2.v20120308.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/lib/servlet-api-3.0.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/build/core/classes/java:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/junit-4.10.jar:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/test-framework/lib/randomizedtesting-runner-2.0.4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-launcher.jar:/home/hudson/.ant/lib/ivy-2.2.0.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-regexp.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bsf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-net.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-antlr.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-log4j.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-commons-logging.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-oro.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-junit4.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-netrexx.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-swing.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-javamail.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jsch.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jmf.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-bcel.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jai.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-testutil.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-apache-resolver.jar:/home/hudson/hudson-slave/tools/Ant/ant-1.8.2/lib/ant-jdepend.jar:/usr/local/openjdk6/lib/tools.jar:/home/hudson/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.0.4.jar
[junit4:junit4]   2> 435 T62 oaz.Environment.logEnv Client environment:java.library.path=/usr/local/openjdk6/jre/lib/amd64/server:/usr/local/openjdk6/jre/lib/amd64:/usr/local/openjdk6/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib:/usr/local/lib
[junit4:junit4]   2> 435 T62 oaz.Environment.logEnv Client environment:java.io.tmpdir=.
[junit4:junit4]   2> 436 T62 oaz.Environment.logEnv Client environment:java.compiler=<NA>
[junit4:junit4]   2> 436 T62 oaz.Environment.logEnv Client environment:os.name=FreeBSD
[junit4:junit4]   2> 436 T62 oaz.Environment.logEnv Client environment:os.arch=amd64
[junit4:junit4]   2> 437 T62 oaz.Environment.logEnv Client environment:os.version=9.0-RELEASE
[junit4:junit4]   2> 437 T62 oaz.Environment.logEnv Client environment:user.name=hudson
[junit4:junit4]   2> 437 T62 oaz.Environment.logEnv Client environment:user.home=/home/hudson
[junit4:junit4]   2> 437 T62 oaz.Environment.logEnv Client environment:user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 445 T62 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63131 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@1d78ff42
[junit4:junit4]   2> 485 T68 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63131
[junit4:junit4]   2> 485 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 489 T68 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63131, initiating session
[junit4:junit4]   2> 490 T64 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:19039
[junit4:junit4]   2> 504 T64 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:19039
[junit4:junit4]   2> 516 T66 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 543 T66 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab16494ff0000 with negotiated timeout 10000 for client /140.211.11.196:19039
[junit4:junit4]   2> 543 T68 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63131, sessionid = 0x13ab16494ff0000, negotiated timeout = 10000
[junit4:junit4]   2> 551 T69 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d78ff42 name:ZooKeeperConnection Watcher:127.0.0.1:63131 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 551 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 554 T62 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 602 T67 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ab16494ff0000
[junit4:junit4]   2> 604 T69 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 604 T62 oaz.ZooKeeper.close Session: 0x13ab16494ff0000 closed
[junit4:junit4]   2> 605 T64 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:19039 which had sessionid 0x13ab16494ff0000
[junit4:junit4]   2> 604 T62 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63131/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@37fa350b
[junit4:junit4]   2> 606 T70 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63131
[junit4:junit4]   2> 606 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 607 T70 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63131, initiating session
[junit4:junit4]   2> 607 T64 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:37431
[junit4:junit4]   2> 607 T64 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:37431
[junit4:junit4]   2> 609 T66 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab16494ff0001 with negotiated timeout 10000 for client /140.211.11.196:37431
[junit4:junit4]   2> 609 T70 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63131, sessionid = 0x13ab16494ff0001, negotiated timeout = 10000
[junit4:junit4]   2> 609 T71 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37fa350b name:ZooKeeperConnection Watcher:127.0.0.1:63131/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 609 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 620 T62 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 625 T62 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 633 T62 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 636 T62 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 641 T62 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 658 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 665 T62 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 666 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 773 T62 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 774 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 787 T62 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 788 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 792 T62 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 792 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 796 T62 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 797 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 801 T62 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 801 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 805 T62 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 806 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 810 T62 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 810 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 814 T62 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 815 T62 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 818 T67 oazs.PrepRequestProcessor.pRequest Processed session termination for sessionid: 0x13ab16494ff0001
[junit4:junit4]   2> 820 T71 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 820 T62 oaz.ZooKeeper.close Session: 0x13ab16494ff0001 closed
[junit4:junit4]   2> 820 T64 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:37431 which had sessionid 0x13ab16494ff0001
[junit4:junit4]   2> 1353 T62 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1586 T62 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:61401
[junit4:junit4]   2> 1610 T62 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1610 T62 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1611 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1638 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1638 T62 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1638 T62 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1639 T62 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1639 T62 oasc.CoreContainer.<init> New CoreContainer 2141355741
[junit4:junit4]   2> 1640 T62 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1640 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1660 T62 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1672 T62 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:63131/solr
[junit4:junit4]   2> 1693 T62 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63131/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@101b8d82
[junit4:junit4]   2> 1694 T82 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63131
[junit4:junit4]   2> 1694 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 1696 T82 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63131, initiating session
[junit4:junit4]   2> 1696 T64 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:62446
[junit4:junit4]   2> 1696 T64 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:62446
[junit4:junit4]   2> 1699 T66 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab16494ff0002 with negotiated timeout 5000 for client /140.211.11.196:62446
[junit4:junit4]   2> 1699 T82 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63131, sessionid = 0x13ab16494ff0002, negotiated timeout = 5000
[junit4:junit4]   2> 1699 T83 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@101b8d82 name:ZooKeeperConnection Watcher:127.0.0.1:63131/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 1700 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 1716 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1724 T62 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 1727 T62 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61401_solr
[junit4:junit4]   2> 1731 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:delete cxid:0x7 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:61401_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:61401_solr
[junit4:junit4]   2> 1732 T62 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:61401_solr
[junit4:junit4]   2> 1736 T62 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1737 T62 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1737 T62 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1737 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1738 T62 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1738 T62 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1738 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1739 T62 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1739 T62 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1739 T62 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1750 T62 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 1766 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:delete cxid:0x15 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 1767 T62 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 1771 T62 oasc.Overseer.start Overseer (id=88578188241731586-127.0.0.1:61401_solr-n_0000000000) starting
[junit4:junit4]   2> 1771 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:create cxid:0x1a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1786 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:create cxid:0x1b zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1787 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:create cxid:0x1c zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1792 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:create cxid:0x1d zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 1800 T85 oasc.OverseerCollectionProcessor.run Process current queue of collection messages
[junit4:junit4]   2> 1811 T62 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 1815 T62 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 1831 T84 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 1858 T62 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1859 T62 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 1860 T62 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 1860 T62 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 1862 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1863 T62 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1864 T62 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1916 T62 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1976 T62 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 2078 T62 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2089 T62 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2521 T62 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2530 T62 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2533 T62 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2551 T62 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2555 T62 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2559 T62 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2561 T62 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2564 T62 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351595889579/control/data/
[junit4:junit4]   2> 2567 T62 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2180b22e
[junit4:junit4]   2> 2591 T62 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351595889579/control/data/index/
[junit4:junit4]   2> 2591 T62 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351595889579/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2592 T62 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351595889579/control/data/index forceNew:false
[junit4:junit4]   2> 2598 T62 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351595889579/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@720bffd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2598 T62 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2599 T62 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 2601 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2601 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 2602 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2602 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 2603 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2603 T62 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2604 T62 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2604 T62 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2605 T62 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2606 T62 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 2606 T62 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 2607 T62 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2607 T62 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 2607 T62 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 2608 T62 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 2608 T62 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2609 T62 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2609 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2610 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2611 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2611 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2612 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2612 T62 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 2613 T62 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 2613 T62 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 2618 T62 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2622 T62 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 2623 T62 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 2627 T62 oass.SolrIndexSearcher.<init> Opening Searcher@d3bef50 main
[junit4:junit4]   2> 2644 T62 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2645 T62 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2645 T62 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 2651 T62 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2707 T90 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d3bef50 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2711 T62 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2712 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:create cxid:0x53 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2724 T62 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2724 T62 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:61401/solr shardId:control_shard
[junit4:junit4]   2> 2734 T62 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 2744 T62 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/control_shard/election
[junit4:junit4]   2> 2763 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:delete cxid:0x62 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2764 T62 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2765 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:create cxid:0x63 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2838 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 2839 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61401_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61401/solr"}
[junit4:junit4]   2> 2849 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:create cxid:0x6a zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2879 T83 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3273 T62 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 3274 T62 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 3275 T62 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:61401/solr/collection1/
[junit4:junit4]   2> 3275 T62 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 3276 T62 oasc.SyncStrategy.syncToMe http://127.0.0.1:61401/solr/collection1/ has no replicas
[junit4:junit4]   2> 3276 T62 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:61401/solr/collection1/
[junit4:junit4]   2> 3277 T62 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/control_shard
[junit4:junit4]   2> 3283 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0002 type:create cxid:0x88 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3388 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3408 T83 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (1)
[junit4:junit4]   2> 3448 T62 oasc.ZkController.register We are http://127.0.0.1:61401/solr/collection1/ and leader is http://127.0.0.1:61401/solr/collection1/
[junit4:junit4]   2> 3452 T62 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:61401/solr
[junit4:junit4]   2> 3453 T62 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3453 T62 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3457 T62 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3458 T62 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3459 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3532 T62 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3719 T62 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 3722 T62 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:39828
[junit4:junit4]   2> 3723 T62 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3724 T62 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351595893132
[junit4:junit4]   2> 3724 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351595893132/'
[junit4:junit4]   2> 3760 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 3761 T62 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3761 T62 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351595893132
[junit4:junit4]   2> 3762 T62 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351595893132/solr.xml
[junit4:junit4]   2> 3762 T62 oasc.CoreContainer.<init> New CoreContainer 1029943554
[junit4:junit4]   2> 3763 T62 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351595893132/'
[junit4:junit4]   2> 3764 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351595893132/'
[junit4:junit4]   2> 3790 T62 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3806 T62 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:63131/solr
[junit4:junit4]   2> 3806 T62 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63131/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@b34bf3
[junit4:junit4]   2> 3807 T100 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63131
[junit4:junit4]   2> 3807 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3809 T100 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63131, initiating session
[junit4:junit4]   2> 3809 T64 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:18064
[junit4:junit4]   2> 3810 T64 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:18064
[junit4:junit4]   2> 3815 T66 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab16494ff0003 with negotiated timeout 5000 for client /140.211.11.196:18064
[junit4:junit4]   2> 3816 T100 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63131, sessionid = 0x13ab16494ff0003, negotiated timeout = 5000
[junit4:junit4]   2> 3816 T101 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b34bf3 name:ZooKeeperConnection Watcher:127.0.0.1:63131/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3816 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3817 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0003 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3830 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0003 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3833 T62 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39828_solr
[junit4:junit4]   2> 3834 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0003 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:39828_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:39828_solr
[junit4:junit4]   2> 3837 T62 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39828_solr
[junit4:junit4]   2> 3845 T83 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 3846 T62 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 3846 T62 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3847 T62 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 3847 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3848 T62 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3848 T62 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3849 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3849 T62 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3850 T62 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3850 T62 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 3874 T62 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3881 T62 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351595893132/collection1
[junit4:junit4]   2> 3881 T62 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 3882 T62 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 3883 T62 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 3885 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351595893132/collection1/'
[junit4:junit4]   2> 3886 T62 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty1-1351595893132/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 3887 T62 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty1-1351595893132/collection1/lib/README' to classloader
[junit4:junit4]   2> 3915 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3916 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard":"control_shard",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:61401_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:61401/solr"}
[junit4:junit4]   2> 3930 T83 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 3930 T101 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 3947 T62 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 4024 T62 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 4126 T62 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 4140 T62 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 4832 T62 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 4864 T62 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 4870 T62 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 4901 T62 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4908 T62 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4914 T62 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4917 T62 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4919 T62 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty1-1351595893132/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty1/
[junit4:junit4]   2> 4920 T62 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2180b22e
[junit4:junit4]   2> 4922 T62 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty1/index/
[junit4:junit4]   2> 4923 T62 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 4924 T62 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty1/index forceNew:false
[junit4:junit4]   2> 4931 T62 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@24c3e952; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4932 T62 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4933 T62 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 4936 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 4936 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 4937 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 4937 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 4938 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 4939 T62 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 4940 T62 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 4941 T62 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 4942 T62 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 4942 T62 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 4943 T62 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 4944 T62 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 4945 T62 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 4945 T62 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 4946 T62 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 4947 T62 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 4947 T62 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4948 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4949 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4950 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4951 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4951 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4952 T62 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 4953 T62 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 4954 T62 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 4961 T62 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 4966 T62 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 4968 T62 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 4973 T62 oass.SolrIndexSearcher.<init> Opening Searcher@3768442e main
[junit4:junit4]   2> 4975 T62 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 4975 T62 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 4976 T62 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 4986 T62 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 5042 T105 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3768442e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5049 T62 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 5442 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5443 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:39828_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:39828/solr"}
[junit4:junit4]   2> 5444 T84 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 5464 T101 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5464 T83 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5553 T62 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 5553 T62 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:39828/solr shardId:shard1
[junit4:junit4]   2> 5555 T62 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 5573 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0003 type:delete cxid:0x43 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 5574 T62 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 5575 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0003 type:create cxid:0x44 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5584 T62 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 5585 T62 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 5585 T62 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39828/solr/collection1/
[junit4:junit4]   2> 5586 T62 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 5586 T62 oasc.SyncStrategy.syncToMe http://127.0.0.1:39828/solr/collection1/ has no replicas
[junit4:junit4]   2> 5586 T62 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39828/solr/collection1/
[junit4:junit4]   2> 5587 T62 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 5592 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0003 type:create cxid:0x4e zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 5970 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 5990 T83 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 5990 T101 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (2)
[junit4:junit4]   2> 6003 T62 oasc.ZkController.register We are http://127.0.0.1:39828/solr/collection1/ and leader is http://127.0.0.1:39828/solr/collection1/
[junit4:junit4]   2> 6003 T62 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39828/solr
[junit4:junit4]   2> 6004 T62 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 6004 T62 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6013 T62 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6014 T62 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 6014 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 6015 T62 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 6130 T62 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 6133 T62 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:13544
[junit4:junit4]   2> 6134 T62 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6134 T62 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351595895600
[junit4:junit4]   2> 6134 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351595895600/'
[junit4:junit4]   2> 6159 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 6159 T62 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 6160 T62 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351595895600
[junit4:junit4]   2> 6160 T62 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351595895600/solr.xml
[junit4:junit4]   2> 6160 T62 oasc.CoreContainer.<init> New CoreContainer 1116940434
[junit4:junit4]   2> 6161 T62 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351595895600/'
[junit4:junit4]   2> 6161 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351595895600/'
[junit4:junit4]   2> 6179 T62 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 6189 T62 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:63131/solr
[junit4:junit4]   2> 6190 T62 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63131/solr sessionTimeout=5000 watcher=org.apache.solr.common.cloud.ConnectionManager@16881734
[junit4:junit4]   2> 6191 T115 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63131
[junit4:junit4]   2> 6191 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 6192 T115 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63131, initiating session
[junit4:junit4]   2> 6192 T64 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:14685
[junit4:junit4]   2> 6193 T64 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:14685
[junit4:junit4]   2> 6195 T66 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab16494ff0004 with negotiated timeout 5000 for client /140.211.11.196:14685
[junit4:junit4]   2> 6195 T115 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63131, sessionid = 0x13ab16494ff0004, negotiated timeout = 5000
[junit4:junit4]   2> 6195 T116 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16881734 name:ZooKeeperConnection Watcher:127.0.0.1:63131/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 6195 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 6196 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0004 type:create cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6211 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0004 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 6212 T62 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13544_solr
[junit4:junit4]   2> 6213 T67 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13ab16494ff0004 type:delete cxid:0x4 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13544_solr Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13544_solr
[junit4:junit4]   2> 6219 T62 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13544_solr
[junit4:junit4]   2> 6222 T83 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 6222 T101 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 6222 T62 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 6223 T62 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 6223 T62 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 6223 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 6224 T62 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 6224 T62 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 6224 T62 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 6225 T62 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 6225 T62 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 6225 T62 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 6239 T62 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 6244 T62 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351595895600/collection1
[junit4:junit4]   2> 6245 T62 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 6246 T62 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 6246 T62 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 6247 T62 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351595895600/collection1/'
[junit4:junit4]   2> 6248 T62 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty2-1351595895600/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 6249 T62 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-jetty2-1351595895600/collection1/lib/README' to classloader
[junit4:junit4]   2> 6285 T62 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 6335 T62 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 6436 T62 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 6447 T62 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6507 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6508 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:39828_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:39828/solr"}
[junit4:junit4]   2> 6522 T116 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6522 T83 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 6522 T101 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7054 T62 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 7066 T62 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 7070 T62 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 7092 T62 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7098 T62 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 7103 T62 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7105 T62 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 7107 T62 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-jetty2-1351595895600/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty2/
[junit4:junit4]   2> 7107 T62 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2180b22e
[junit4:junit4]   2> 7109 T62 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty2/index/
[junit4:junit4]   2> 7110 T62 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 7111 T62 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty2/index forceNew:false
[junit4:junit4]   2> 7123 T62 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1976bf9e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 7123 T62 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 7124 T62 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 7127 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 7128 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 7128 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 7129 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 7130 T62 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 7131 T62 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 7131 T62 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 7132 T62 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 7133 T62 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 7134 T62 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 7135 T62 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 7136 T62 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 7136 T62 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 7137 T62 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 7138 T62 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 7138 T62 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 7139 T62 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7140 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7141 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7142 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7143 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7144 T62 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7144 T62 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 7145 T62 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 7146 T62 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 7153 T62 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 7158 T62 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 7160 T62 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: [SCHEMA.XML, OLD_SYNONYMS.TXT, STOPWORDS.TXT, PROTWORDS.TXT, OPEN-EXCHANGE-RATES.JSON, SYNONYMS.TXT, CURRENCY.XML, MAPPING-ISOLATIN1ACCENT.TXT]
[junit4:junit4]   2> 7164 T62 oass.SolrIndexSearcher.<init> Opening Searcher@79ddd026 main
[junit4:junit4]   2> 7165 T62 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 7166 T62 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 7167 T62 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 7175 T62 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 7226 T121 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79ddd026 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 7233 T62 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7531 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7533 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:13544_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:13544/solr"}
[junit4:junit4]   2> 7547 T83 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7547 T101 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7547 T116 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 7737 T62 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 7737 T62 oasc.ZkController.register Register shard - core:collection1 address:http://127.0.0.1:13544/solr shardId:shard1
[junit4:junit4]   2> 7742 T62 oasc.ZkController.register We are http://127.0.0.1:13544/solr/collection1/ and leader is http://127.0.0.1:39828/solr/collection1/
[junit4:junit4]   2> 7743 T62 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13544/solr
[junit4:junit4]   2> 7743 T62 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 7744 T62 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C11 name=collection1 org.apache.solr.core.SolrCore@229ff6d1 url=http://127.0.0.1:13544/solr/collection1 node=127.0.0.1:13544_solr
[junit4:junit4]   2> 7751 T122 C11 P13544 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 7752 T122 C11 P13544 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 7752 T62 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7752 T122 C11 P13544 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7754 T62 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 7755 T122 C11 P13544 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:39828/solr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 7755 T62 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 7756 T62 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 7765 T122 C11 P13544 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 7776 T62 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63131/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@644b27ad
[junit4:junit4]   2> 7777 T123 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63131
[junit4:junit4]   2> 7777 T62 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 7778 T123 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63131, initiating session
[junit4:junit4]   2> 7779 T64 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:23942
[junit4:junit4]   2> 7779 T64 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:23942
[junit4:junit4]   2> 7781 T66 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab16494ff0005 with negotiated timeout 10000 for client /140.211.11.196:23942
[junit4:junit4]   2> 7781 T123 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63131, sessionid = 0x13ab16494ff0005, negotiated timeout = 10000
[junit4:junit4]   2> 7781 T124 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@644b27ad name:ZooKeeperConnection Watcher:127.0.0.1:63131/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 7782 T62 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 7783 T62 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 7783 T122 C11 P13544 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:13544/solr START replicas=[http://127.0.0.1:39828/solr/collection1/] nUpdates=100
[junit4:junit4]   2> 7790 T62 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 7800 T62 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 7801 T122 C11 P13544 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 7802 T122 C11 P13544 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 7802 T122 C11 P13544 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 7802 T122 C11 P13544 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 7846 T62 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8053 T84 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 8054 T84 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:13544_solr",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:13544/solr"}
[junit4:junit4]   2> 8069 T83 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8069 T101 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8069 T124 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> 8069 T116 oascc.ZkStateReader$2.process A cluster state change has occurred - updating... (3)
[junit4:junit4]   2> ASYNC  NEW_CORE C12 name=collection1 org.apache.solr.core.SolrCore@63acd9bf url=http://127.0.0.1:39828/solr/collection1 node=127.0.0.1:39828_solr
[junit4:junit4]   2> 9284 T102 C12 P39828 REQ /get {distrib=false&qt=/get&wt=javabin&version=2&getVersions=100} status=0 QTime=18 
[junit4:junit4]   2> ASYNC  NEW_CORE C13 name=collection1 org.apache.solr.core.SolrCore@3f1a2258 url=http://127.0.0.1:61401/solr/collection1 node=127.0.0.1:61401_solr
[junit4:junit4]   2> 9355 T88 C13 P61401 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16
[junit4:junit4]   2> 9422 T88 C13 P61401 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351595889579/control/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@720bffd; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9423 T88 C13 P61401 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 9516 T88 C13 P61401 /update {wt=javabin&version=2} {add=[1 (1417251021381435393)]} 0 179
[junit4:junit4]   2> 9522 T89 C13 P61401 /update {wt=javabin&version=2} {add=[10001 (1417251021381435392)]} 0 185
[junit4:junit4]   2> 9522 T126 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:63131/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@4c27d525
[junit4:junit4]   2> 9524 T126 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 9524 T129 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server /127.0.0.1:63131
[junit4:junit4]   2> 9526 T129 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:63131, initiating session
[junit4:junit4]   2> 9526 T64 oazs.NIOServerCnxn$Factory.run Accepted socket connection from /140.211.11.196:32678
[junit4:junit4]   2> 9527 T64 oazs.NIOServerCnxn.readConnectRequest Client attempting to establish new session at /140.211.11.196:32678
[junit4:junit4]   2> 9529 T66 oazs.NIOServerCnxn.finishSessionInit Established session 0x13ab16494ff0006 with negotiated timeout 10000 for client /140.211.11.196:32678
[junit4:junit4]   2> 9529 T129 oaz.ClientCnxn$SendThread.readConnectResult Session establishment complete on server localhost/127.0.0.1:63131, sessionid = 0x13ab16494ff0006, negotiated timeout = 10000
[junit4:junit4]   2> 9530 T130 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4c27d525 name:ZooKeeperConnection Watcher:127.0.0.1:63131/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 9530 T126 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 9531 T126 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9559 T103 C12 P39828 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@24c3e952; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9559 T103 C12 P39828 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2>  C11_STATE=coll:collection1 core:collection1 props:{shard=shard1, roles=null, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:13544_solr, base_url=http://127.0.0.1:13544/solr}
[junit4:junit4]   2> 9619 T119 C11 P13544 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=NRTCachingDirectory(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.RecoveryZkTest-1351595889579/jetty2/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@1976bf9e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9620 T119 C11 P13544 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 9625 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=11 
[junit4:junit4]   2> 9625 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=10 
[junit4:junit4]   2> 9626 T103 C12 P39828 /update {wt=javabin&version=2} {add=[10001 (1417251021526138880)]} 0 79
[junit4:junit4]   2> 9627 T102 C12 P39828 /update {wt=javabin&version=2} {add=[1 (1417251021526138881)]} 0 80
[junit4:junit4]   2> 9633 T89 C13 P61401 /update {wt=javabin&version=2} {add=[10002 (1417251021606879232)]} 0 2
[junit4:junit4]   2> 9635 T87 C13 P61401 /update {wt=javabin&version=2} {add=[2 (1417251021606879233)]} 0 4
[junit4:junit4]   2> 9648 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9649 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10002 (1417251021613170688)]} 0 11
[junit4:junit4]   2> 9649 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9650 T103 C12 P39828 /update {wt=javabin&version=2} {add=[2 (1417251021615267840)]} 0 11
[junit4:junit4]   2> 9653 T88 C13 P61401 /update {wt=javabin&version=2} {delete=[10002 (-1417251021628899328)]} 0 0
[junit4:junit4]   2> 9655 T89 C13 P61401 /update {wt=javabin&version=2} {delete=[2 (-1417251021630996480)]} 0 1
[junit4:junit4]   2> 9665 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9666 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 9666 T128 C12 P39828 /update {wt=javabin&version=2} {delete=[2 (-1417251021635190784)]} 0 7
[junit4:junit4]   2> 9667 T102 C12 P39828 /update {wt=javabin&version=2} {delete=[10002 (-1417251021633093632)]} 0 10
[junit4:junit4]   2> 9672 T87 C13 P61401 /update {wt=javabin&version=2} {add=[3 (1417251021646725120)]} 0 3
[junit4:junit4]   2> 9674 T88 C13 P61401 /update {wt=javabin&version=2} {add=[10003 (1417251021647773696)]} 0 4
[junit4:junit4]   2> 9683 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9684 T103 C12 P39828 /update {wt=javabin&version=2} {add=[3 (1417251021653016576)]} 0 9
[junit4:junit4]   2> 9685 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9686 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10003 (1417251021655113728)]} 0 9
[junit4:junit4]   2> 9688 T89 C13 P61401 /update {wt=javabin&version=2} {delete=[3 (-1417251021665599488)]} 0 1
[junit4:junit4]   2> 9690 T87 C13 P61401 /update {wt=javabin&version=2} {delete=[10003 (-1417251021667696640)]} 0 0
[junit4:junit4]   2> 9695 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9696 T102 C12 P39828 /update {wt=javabin&version=2} {delete=[3 (-1417251021669793792)]} 0 5
[junit4:junit4]   2> 9697 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9698 T103 C12 P39828 /update {wt=javabin&version=2} {delete=[10003 (-1417251021671890944)]} 0 5
[junit4:junit4]   2> 9702 T88 C13 P61401 /update {wt=javabin&version=2} {add=[4 (1417251021678182400)]} 0 2
[junit4:junit4]   2> 9704 T89 C13 P61401 /update {wt=javabin&version=2} {add=[10004 (1417251021680279552)]} 0 3
[junit4:junit4]   2> 9713 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9714 T128 C12 P39828 /update {wt=javabin&version=2} {add=[4 (1417251021684473856)]} 0 8
[junit4:junit4]   2> 9715 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9716 T102 C12 P39828 /update {wt=javabin&version=2} {add=[10004 (1417251021686571008)]} 0 9
[junit4:junit4]   2> 9721 T87 C13 P61401 /update {wt=javabin&version=2} {add=[5 (1417251021698105344)]} 0 3
[junit4:junit4]   2> 9747 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=9 
[junit4:junit4]   2> 9748 T103 C12 P39828 /update {wt=javabin&version=2} {add=[5 (1417251021708591104)]} 0 23
[junit4:junit4]   2> 9756 T89 C13 P61401 /update {wt=javabin&version=2} {add=[6 (1417251021732708352)]} 0 4
[junit4:junit4]   2> 9757 T88 C13 P61401 /update {wt=javabin&version=2} {add=[10005 (1417251021699153920)]} 0 37
[junit4:junit4]   2> 9771 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9772 T103 C12 P39828 /update {wt=javabin&version=2} {add=[10005 (1417251021742145536)]} 0 12
[junit4:junit4]   2> 9778 T87 C13 P61401 /update {wt=javabin&version=2} {add=[10006 (1417251021757874176)]} 0 3
[junit4:junit4]   2> 9783 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9784 T102 C12 P39828 /update {wt=javabin&version=2} {add=[6 (1417251021741096960)]} 0 25
[junit4:junit4]   2> 9790 T89 C13 P61401 /update {wt=javabin&version=2} {add=[7 (1417251021770457088)]} 0 2
[junit4:junit4]   2> 9790 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9791 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10006 (1417251021764165632)]} 0 9
[junit4:junit4]   2> 9798 T88 C13 P61401 /update {wt=javabin&version=2} {add=[10007 (1417251021777797120)]} 0 3
[junit4:junit4]   2> 9801 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9802 T103 C12 P39828 /update {wt=javabin&version=2} {add=[7 (1417251021776748544)]} 0 9
[junit4:junit4]   2> 9808 T87 C13 P61401 /update {wt=javabin&version=2} {add=[8 (1417251021789331456)]} 0 2
[junit4:junit4]   2> 9819 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9820 T128 C12 P39828 /update {wt=javabin&version=2} {add=[8 (1417251021795622912)]} 0 9
[junit4:junit4]   2> 9824 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=17 
[junit4:junit4]   2> 9826 T89 C13 P61401 /update {wt=javabin&version=2} {add=[9 (1417251021808205824)]} 0 3
[junit4:junit4]   2> 9826 T102 C12 P39828 /update {wt=javabin&version=2} {add=[10007 (1417251021785137152)]} 0 25
[junit4:junit4]   2> 9832 T88 C13 P61401 /update {wt=javabin&version=2} {add=[10008 (1417251021814497280)]} 0 2
[junit4:junit4]   2> 9839 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9841 T103 C12 P39828 /update {wt=javabin&version=2} {add=[9 (1417251021815545856)]} 0 11
[junit4:junit4]   2> 9843 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9844 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10008 (1417251021820788736)]} 0 8
[junit4:junit4]   2> 9847 T87 C13 P61401 /update {wt=javabin&version=2} {add=[10 (1417251021830225920)]} 0 3
[junit4:junit4]   2> 9853 T89 C13 P61401 /update {wt=javabin&version=2} {add=[10009 (1417251021835468800)]} 0 4
[junit4:junit4]   2> 9859 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9860 T103 C12 P39828 /update {wt=javabin&version=2} {add=[10 (1417251021837565952)]} 0 9
[junit4:junit4]   2> 9864 T88 C13 P61401 /update {wt=javabin&version=2} {delete=[6 (-1417251021850148864)]} 0 0
[junit4:junit4]   2> 9864 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9866 T102 C12 P39828 /update {wt=javabin&version=2} {add=[10009 (1417251021842808832)]} 0 10
[junit4:junit4]   2> 9873 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 9874 T128 C12 P39828 /update {wt=javabin&version=2} {delete=[6 (-1417251021855391744)]} 0 5
[junit4:junit4]   2> 9882 T89 C13 P61401 /update {wt=javabin&version=2} {add=[11 (1417251021864828928)]} 0 4
[junit4:junit4]   2> 9888 T87 C13 P61401 /update {wt=javabin&version=2} {add=[10010 (1417251021856440320)]} 0 18
[junit4:junit4]   2> 9901 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9901 T102 C12 P39828 /update {wt=javabin&version=2} {add=[10010 (1417251021879508992)]} 0 9
[junit4:junit4]   2> 9906 T88 C13 P61401 /update {wt=javabin&version=2} {delete=[10006 (-1417251021894189056)]} 0 1
[junit4:junit4]   2> 9910 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9911 T103 C12 P39828 /update {wt=javabin&version=2} {add=[11 (1417251021873217536)]} 0 25
[junit4:junit4]   2> 9916 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 9917 T128 C12 P39828 /update {wt=javabin&version=2} {delete=[10006 (-1417251021899431936)]} 0 6
[junit4:junit4]   2> 9920 T87 C13 P61401 /update {wt=javabin&version=2} {add=[12 (1417251021904674816)]} 0 5
[junit4:junit4]   2> 9925 T89 C13 P61401 /update {wt=javabin&version=2} {add=[10011 (1417251021912014848)]} 0 2
[junit4:junit4]   2> 9931 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9932 T102 C12 P39828 /update {wt=javabin&version=2} {add=[12 (1417251021913063424)]} 0 8
[junit4:junit4]   2> 9938 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9938 T87 C13 P61401 /update {wt=javabin&version=2} {add=[13 (1417251021925646336)]} 0 2
[junit4:junit4]   2> 9939 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10011 (1417251021918306304)]} 0 10
[junit4:junit4]   2> 9944 T88 C13 P61401 /update {wt=javabin&version=2} {add=[10012 (1417251021932986368)]} 0 2
[junit4:junit4]   2> 9949 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9950 T103 C12 P39828 /update {wt=javabin&version=2} {add=[13 (1417251021931937792)]} 0 9
[junit4:junit4]   2> 9955 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9955 T89 C13 P61401 /update {wt=javabin&version=2} {add=[14 (1417251021944520704)]} 0 2
[junit4:junit4]   2> 9956 T102 C12 P39828 /update {wt=javabin&version=2} {add=[10012 (1417251021938229248)]} 0 9
[junit4:junit4]   2> 9962 T87 C13 P61401 /update {wt=javabin&version=2} {add=[10013 (1417251021950812160)]} 0 2
[junit4:junit4]   2> 9967 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9969 T128 C12 P39828 /update {wt=javabin&version=2} {add=[14 (1417251021950812160)]} 0 9
[junit4:junit4]   2> 9974 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 9975 T103 C12 P39828 /update {wt=javabin&version=2} {add=[10013 (1417251021957103616)]} 0 9
[junit4:junit4]   2> 9976 T88 C13 P61401 /update {wt=javabin&version=2} {add=[15 (1417251021965492224)]} 0 3
[junit4:junit4]   2> 9980 T89 C13 P61401 /update {wt=javabin&version=2} {add=[10014 (1417251021970735104)]} 0 2
[junit4:junit4]   2> 9992 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 9993 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10014 (1417251021975977984)]} 0 10
[junit4:junit4]   2> 10006 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=21 
[junit4:junit4]   2> 10007 T102 C12 P39828 /update {wt=javabin&version=2} {add=[15 (1417251021971783680)]} 0 28
[junit4:junit4]   2> 10011 T88 C13 P61401 /update {wt=javabin&version=2} {delete=[8 (-1417251022004289536)]} 0 0
[junit4:junit4]   2> 10017 T87 C13 P61401 /update {wt=javabin&version=2} {add=[10015 (1417251021989609472)]} 0 21
[junit4:junit4]   2> 10018 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10019 T103 C12 P39828 /update {wt=javabin&version=2} {delete=[8 (-1417251022008483840)]} 0 4
[junit4:junit4]   2> 10027 T89 C13 P61401 /update {wt=javabin&version=2} {add=[16 (1417251022016872448)]} 0 4
[junit4:junit4]   2> 10040 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10041 T102 C12 P39828 /update {wt=javabin&version=2} {add=[16 (1417251022025261056)]} 0 11
[junit4:junit4]   2> 10042 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10043 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10015 (1417251022014775296)]} 0 23
[junit4:junit4]   2> 10045 T88 C13 P61401 /update {wt=javabin&version=2} {delete=[9 (-1417251022039941120)]} 0 0
[junit4:junit4]   2> 10047 T87 C13 P61401 /update {wt=javabin&version=2} {delete=[10008 (-1417251022042038272)]} 0 1
[junit4:junit4]   2> 10052 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10053 T103 C12 P39828 /update {wt=javabin&version=2} {delete=[9 (-1417251022044135424)]} 0 5
[junit4:junit4]   2> 10054 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10055 T102 C12 P39828 /update {wt=javabin&version=2} {delete=[10008 (-1417251022046232576)]} 0 5
[junit4:junit4]   2> 10058 T89 C13 P61401 /update {wt=javabin&version=2} {add=[17 (1417251022052524032)]} 0 2
[junit4:junit4]   2> 10061 T88 C13 P61401 /update {wt=javabin&version=2} {add=[10016 (1417251022054621184)]} 0 3
[junit4:junit4]   2> 10069 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10070 T128 C12 P39828 /update {wt=javabin&version=2} {add=[17 (1417251022057766912)]} 0 9
[junit4:junit4]   2> 10072 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10073 T103 C12 P39828 /update {wt=javabin&version=2} {add=[10016 (1417251022060912640)]} 0 8
[junit4:junit4]   2> 10074 T87 C13 P61401 /update {wt=javabin&version=2} {delete=[11 (-1417251022070349824)]} 0 0
[junit4:junit4]   2> 10077 T89 C13 P61401 /update {wt=javabin&version=2} {delete=[10009 (-1417251022073495552)]} 0 1
[junit4:junit4]   2> 10081 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10083 T102 C12 P39828 /update {wt=javabin&version=2} {delete=[11 (-1417251022074544128)]} 0 6
[junit4:junit4]   2> 10085 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10086 T128 C12 P39828 /update {wt=javabin&version=2} {delete=[10009 (-1417251022077689856)]} 0 5
[junit4:junit4]   2> 10089 T88 C13 P61401 /update {wt=javabin&version=2} {add=[18 (1417251022083981312)]} 0 3
[junit4:junit4]   2> 10092 T87 C13 P61401 /update {wt=javabin&version=2} {add=[10017 (1417251022087127040)]} 0 2
[junit4:junit4]   2> 10100 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10101 T103 C12 P39828 /update {wt=javabin&version=2} {add=[18 (1417251022090272768)]} 0 9
[junit4:junit4]   2> 10105 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10106 T89 C13 P61401 /update {wt=javabin&version=2} {delete=[15 (-1417251022102855680)]} 0 1
[junit4:junit4]   2> 10106 T102 C12 P39828 /update {wt=javabin&version=2} {add=[10017 (1417251022094467072)]} 0 9
[junit4:junit4]   2> 10110 T88 C13 P61401 /update {wt=javabin&version=2} {delete=[10011 (-1417251022108098560)]} 0 1
[junit4:junit4]   2> 10113 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10115 T128 C12 P39828 /update {wt=javabin&version=2} {delete=[15 (-1417251022108098560)]} 0 6
[junit4:junit4]   2> 10118 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10119 T103 C12 P39828 /update {wt=javabin&version=2} {delete=[10011 (-1417251022113341440)]} 0 5
[junit4:junit4]   2> 10122 T87 C13 P61401 /update {wt=javabin&version=2} {add=[19 (1417251022118584320)]} 0 2
[junit4:junit4]   2> 10125 T89 C13 P61401 /update {wt=javabin&version=2} {add=[10018 (1417251022121730048)]} 0 3
[junit4:junit4]   2> 10136 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10137 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10018 (1417251022128021504)]} 0 9
[junit4:junit4]   2> 10141 T88 C13 P61401 /update {wt=javabin&version=2} {delete=[10015 (-1417251022140604416)]} 0 1
[junit4:junit4]   2> 10148 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10149 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=18 
[junit4:junit4]   2> 10149 T103 C12 P39828 /update {wt=javabin&version=2} {delete=[10015 (-1417251022144798720)]} 0 5
[junit4:junit4]   2> 10150 T102 C12 P39828 /update {wt=javabin&version=2} {add=[19 (1417251022124875776)]} 0 24
[junit4:junit4]   2> 10154 T89 C13 P61401 /update {wt=javabin&version=2} {delete=[16 (-1417251022154235905)]} 0 0
[junit4:junit4]   2> 10156 T87 C13 P61401 /update {wt=javabin&version=2} {add=[10019 (1417251022154235904)]} 0 3
[junit4:junit4]   2> 10161 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10162 T103 C12 P39828 /update {wt=javabin&version=2} {delete=[16 (-1417251022158430208)]} 0 5
[junit4:junit4]   2> 10167 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10167 T88 C13 P61401 /update {wt=javabin&version=2} {add=[20 (1417251022166818816)]} 0 1
[junit4:junit4]   2> 10168 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10019 (1417251022160527360)]} 0 9
[junit4:junit4]   2> 10172 T89 C13 P61401 /update {wt=javabin&version=2} {delete=[10016 (-1417251022173110272)]} 0 1
[junit4:junit4]   2> 10179 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10180 T102 C12 P39828 /update {wt=javabin&version=2} {delete=[10016 (-1417251022177304576)]} 0 5
[junit4:junit4]   2> 10180 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10181 T103 C12 P39828 /update {wt=javabin&version=2} {add=[20 (1417251022173110272)]} 0 10
[junit4:junit4]   2> 10190 T88 C13 P61401 /update {wt=javabin&version=2} {add=[21 (1417251022186741760)]} 0 5
[junit4:junit4]   2> 10204 T87 C13 P61401 /update {wt=javabin&version=2} {add=[10020 (1417251022185693184)]} 0 21
[junit4:junit4]   2> 10217 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10218 T102 C12 P39828 /update {wt=javabin&version=2} {add=[10020 (1417251022210859008)]} 0 10
[junit4:junit4]   2> 10224 T89 C13 P61401 /update {wt=javabin&version=2} {add=[10021 (1417251022225539072)]} 0 2
[junit4:junit4]   2> 10233 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=17 
[junit4:junit4]   2> 10234 T128 C12 P39828 /update {wt=javabin&version=2} {add=[21 (1417251022197227520)]} 0 39
[junit4:junit4]   2> 10236 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10237 T103 C12 P39828 /update {wt=javabin&version=2} {add=[10021 (1417251022231830528)]} 0 9
[junit4:junit4]   2> 10238 T88 C13 P61401 /update {wt=javabin&version=2} {delete=[19 (-1417251022242316288)]} 0 0
[junit4:junit4]   2> 10241 T87 C13 P61401 /update {wt=javabin&version=2} {delete=[10019 (-1417251022245462016)]} 0 0
[junit4:junit4]   2> 10246 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10247 T102 C12 P39828 /update {wt=javabin&version=2} {delete=[19 (-1417251022247559168)]} 0 5
[junit4:junit4]   2> 10248 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10249 T103 C12 P39828 /update {wt=javabin&version=2} {delete=[10019 (-1417251022249656320)]} 0 5
[junit4:junit4]   2> 10254 T89 C13 P61401 /update {wt=javabin&version=2} {add=[22 (1417251022256996352)]} 0 2
[junit4:junit4]   2> 10255 T88 C13 P61401 /update {wt=javabin&version=2} {add=[10022 (1417251022258044928)]} 0 3
[junit4:junit4]   2> 10265 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10265 T128 C12 P39828 /update {wt=javabin&version=2} {add=[22 (1417251022263287808)]} 0 8
[junit4:junit4]   2> 10268 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=4 
[junit4:junit4]   2> 10269 T102 C12 P39828 /update {wt=javabin&version=2} {add=[10022 (1417251022265384960)]} 0 10
[junit4:junit4]   2> 10271 T87 C13 P61401 /update {wt=javabin&version=2} {add=[23 (1417251022275870720)]} 0 2
[junit4:junit4]   2> 10275 T89 C13 P61401 /update {wt=javabin&version=2} {add=[10023 (1417251022279016448)]} 0 3
[junit4:junit4]   2> 10282 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10283 T103 C12 P39828 /update {wt=javabin&version=2} {add=[23 (1417251022281113600)]} 0 9
[junit4:junit4]   2> 10285 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10286 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10023 (1417251022285307904)]} 0 8
[junit4:junit4]   2> 10289 T88 C13 P61401 /update {wt=javabin&version=2} {add=[24 (1417251022293696512)]} 0 3
[junit4:junit4]   2> 10291 T87 C13 P61401 /update {wt=javabin&version=2} {add=[10024 (1417251022296842240)]} 0 2
[junit4:junit4]   2> 10300 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10300 T102 C12 P39828 /update {wt=javabin&version=2} {add=[24 (1417251022299987968)]} 0 8
[junit4:junit4]   2> 10303 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10303 T103 C12 P39828 /update {wt=javabin&version=2} {add=[10024 (1417251022303133696)]} 0 8
[junit4:junit4]   2> 10305 T89 C13 P61401 /update {wt=javabin&version=2} {delete=[20 (-1417251022312570880)]} 0 1
[junit4:junit4]   2> 10307 T88 C13 P61401 /update {wt=javabin&version=2} {delete=[10020 (-1417251022314668032)]} 0 1
[junit4:junit4]   2> 10312 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10313 T128 C12 P39828 /update {wt=javabin&version=2} {delete=[20 (-1417251022316765184)]} 0 5
[junit4:junit4]   2> 10314 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=1 
[junit4:junit4]   2> 10315 T102 C12 P39828 /update {wt=javabin&version=2} {delete=[10020 (-1417251022317813760)]} 0 5
[junit4:junit4]   2> 10319 T87 C13 P61401 /update {wt=javabin&version=2} {add=[25 (1417251022325153792)]} 0 3
[junit4:junit4]   2> 10329 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10330 T103 C12 P39828 /update {wt=javabin&version=2} {add=[25 (1417251022331445248)]} 0 8
[junit4:junit4]   2> 10336 T89 C13 P61401 /update {wt=javabin&version=2} {add=[10025 (1417251022327250944)]} 0 18
[junit4:junit4]   2> 10337 T88 C13 P61401 /update {wt=javabin&version=2} {add=[26 (1417251022342979584)]} 0 4
[junit4:junit4]   2> 10350 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10351 T128 C12 P39828 /update {wt=javabin&version=2} {add=[26 (1417251022350319616)]} 0 11
[junit4:junit4]   2> 10355 T89 C13 P61401 /update {wt=javabin&version=2} {delete=[21 (-1417251022364999680)]} 0 1
[junit4:junit4]   2> 10361 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10361 T102 C12 P39828 /update {wt=javabin&version=2} {add=[10025 (1417251022349271040)]} 0 22
[junit4:junit4]   2> 10363 T118 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=0 
[junit4:junit4]   2> 10364 T103 C12 P39828 /update {wt=javabin&version=2} {delete=[21 (-1417251022369193984)]} 0 5
[junit4:junit4]   2> 10367 T87 C13 P61401 /update {wt=javabin&version=2} {add=[10026 (1417251022375485440)]} 0 3
[junit4:junit4]   2> 10370 T88 C13 P61401 /update {wt=javabin&version=2} {add=[27 (1417251022378631168)]} 0 3
[junit4:junit4]   2> 10377 T120 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=3 
[junit4:junit4]   2> 10378 T128 C12 P39828 /update {wt=javabin&version=2} {add=[10026 (1417251022381776896)]} 0 8
[junit4:junit4]   2> 10381 T89 C13 P61401 /update {wt=javabin&version=2} {delete=[10021 (-1417251022392262656)]} 0 0
[junit4:junit4]   2> 10381 T119 C11 P13544 REQ /update {distrib.from=http://127.0.0.1:39828/solr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 10382 T102 C12 P39828 /update {wt=javabin&version=2} {add=[27 (1417251022384922624)]} 0 9
[junit4:junit4]   2> 10388 T87 C13 P61401 /update {wt=javabin&version=2} {add=[28 (1417251022397505536)]} 0 2
[junit4:

[...truncated too long message...]

eptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=168, name=qtp411693457-168 Acceptor1 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=170, name=qtp411693457-170 Acceptor3 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([20E679A9BD29B952]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=79, name=qtp64462389-79 Acceptor1 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=81, name=qtp64462389-81 Acceptor3 SelectChannelConnector@0.0.0.0:61401, state=RUNNABLE, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=80, name=qtp64462389-80 Acceptor2 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=78, name=qtp64462389-78 Acceptor0 SelectChannelConnector@0.0.0.0:61401, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    5) Thread[id=169, name=qtp411693457-169 Acceptor2 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    6) Thread[id=167, name=qtp411693457-167 Acceptor0 SelectChannelConnector@0.0.0.0:13544, state=RUNNABLE, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    7) Thread[id=168, name=qtp411693457-168 Acceptor1 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    8) Thread[id=170, name=qtp411693457-170 Acceptor3 SelectChannelConnector@0.0.0.0:13544, state=BLOCKED, group=TGRP-RecoveryZkTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([20E679A9BD29B952]:0)
[junit4:junit4] Completed on J1 in 87.51s, 1 test, 2 errors <<< FAILURES!

[...truncated 131 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1159: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 815 ignored

Total time: 21 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 820 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/820/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
64 threads leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandler:     1) Thread[id=289, name=qtp968634387-289 Acceptor0 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=99, name=qtp1455364682-99 Acceptor1 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=245, name=qtp1571259428-245 Acceptor1 SelectChannelConnector@0.0.0.0:63180, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=59, name=qtp1164548552-59 Acceptor2 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=272, name=qtp955386212-272 Acceptor1 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=148, name=qtp1566364831-148 Acceptor0 SelectChannelConnector@0.0.0.0:17939, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=48, name=qtp1220869853-48 Acceptor2 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=58, name=qtp1164548552-58 Acceptor1 SelectChannelConnector@0.0.0.0:57149, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=91, name=qtp1396567385-91 Acceptor3 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=47, name=qtp1220869853-47 Acceptor1 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=191, name=qtp1114740402-191 Acceptor2 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=115, name=qtp1594811832-115 Acceptor1 SelectChannelConnector@0.0.0.0:20339, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=246, name=qtp1571259428-246 Acceptor2 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=149, name=qtp1566364831-149 Acceptor1 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=88, name=qtp1396567385-88 Acceptor0 SelectChannelConnector@0.0.0.0:57148, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=32, name=qtp137322702-32 Acceptor0 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=213, name=qtp221024277-213 Acceptor3 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=101, name=qtp1455364682-101 Acceptor3 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=135, name=qtp435252846-135 Acceptor1 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=200, name=qtp1255434575-200 Acceptor0 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=57, name=qtp1164548552-57 Acceptor0 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=89, name=qtp1396567385-89 Acceptor1 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=116, name=qtp1594811832-116 Acceptor2 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=201, name=qtp1255434575-201 Acceptor1 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=247, name=qtp1571259428-247 Acceptor3 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=211, name=qtp221024277-211 Acceptor1 SelectChannelConnector@0.0.0.0:60070, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=203, name=qtp1255434575-203 Acceptor3 SelectChannelConnector@0.0.0.0:13276, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=114, name=qtp1594811832-114 Acceptor0 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=274, name=qtp955386212-274 Acceptor3 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=192, name=qtp1114740402-192 Acceptor3 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=90, name=qtp1396567385-90 Acceptor2 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=273, name=qtp955386212-273 Acceptor2 SelectChannelConnector@0.0.0.0:21766, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=271, name=qtp955386212-271 Acceptor0 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=189, name=qtp1114740402-189 Acceptor0 SelectChannelConnector@0.0.0.0:38544, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=151, name=qtp1566364831-151 Acceptor3 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=100, name=qtp1455364682-100 Acceptor2 SelectChannelConnector@0.0.0.0:38328, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   37) Thread[id=33, name=qtp137322702-33 Acceptor1 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   38) Thread[id=210, name=qtp221024277-210 Acceptor0 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   39) Thread[id=73, name=qtp1342965186-73 Acceptor2 SelectChannelConnector@0.0.0.0:55023, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   40) Thread[id=136, name=qtp435252846-136 Acceptor2 SelectChannelConnector@0.0.0.0:56846, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   41) Thread[id=134, name=qtp435252846-134 Acceptor0 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   42) Thread[id=290, name=qtp968634387-290 Acceptor1 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   43) Thread[id=202, name=qtp1255434575-202 Acceptor2 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   44) Thread[id=117, name=qtp1594811832-117 Acceptor3 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   45) Thread[id=35, name=qtp137322702-35 Acceptor3 SelectChannelConnector@0.0.0.0:17417, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   46) Thread[id=292, name=qtp968634387-292 Acceptor3 SelectChannelConnector@0.0.0.0:34474, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   47) Thread[id=34, name=qtp137322702-34 Acceptor2 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   48) Thread[id=230, name=qtp1500381475-230 Acceptor0 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   49) Thread[id=233, name=qtp1500381475-233 Acceptor3 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   50) Thread[id=212, name=qtp221024277-212 Acceptor2 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   51) Thread[id=244, name=qtp1571259428-244 Acceptor0 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   52) Thread[id=291, name=qtp968634387-291 Acceptor2 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   53) Thread[id=232, name=qtp1500381475-232 Acceptor2 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   54) Thread[id=49, name=qtp1220869853-49 Acceptor3 SelectChannelConnector@0.0.0.0:28190, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   55) Thread[id=74, name=qtp1342965186-74 Acceptor3 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   56) Thread[id=137, name=qtp435252846-137 Acceptor3 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   57) Thread[id=71, name=qtp1342965186-71 Acceptor0 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   58) Thread[id=46, name=qtp1220869853-46 Acceptor0 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   59) Thread[id=231, name=qtp1500381475-231 Acceptor1 SelectChannelConnector@0.0.0.0:30272, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   60) Thread[id=190, name=qtp1114740402-190 Acceptor1 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   61) Thread[id=98, name=qtp1455364682-98 Acceptor0 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   62) Thread[id=150, name=qtp1566364831-150 Acceptor2 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   63) Thread[id=60, name=qtp1164548552-60 Acceptor3 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   64) Thread[id=72, name=qtp1342965186-72 Acceptor1 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 64 threads leaked from SUITE scope at org.apache.solr.handler.TestReplicationHandler: 
   1) Thread[id=289, name=qtp968634387-289 Acceptor0 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=99, name=qtp1455364682-99 Acceptor1 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=245, name=qtp1571259428-245 Acceptor1 SelectChannelConnector@0.0.0.0:63180, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=59, name=qtp1164548552-59 Acceptor2 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=272, name=qtp955386212-272 Acceptor1 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=148, name=qtp1566364831-148 Acceptor0 SelectChannelConnector@0.0.0.0:17939, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=48, name=qtp1220869853-48 Acceptor2 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=58, name=qtp1164548552-58 Acceptor1 SelectChannelConnector@0.0.0.0:57149, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=91, name=qtp1396567385-91 Acceptor3 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=47, name=qtp1220869853-47 Acceptor1 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=191, name=qtp1114740402-191 Acceptor2 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=115, name=qtp1594811832-115 Acceptor1 SelectChannelConnector@0.0.0.0:20339, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=246, name=qtp1571259428-246 Acceptor2 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=149, name=qtp1566364831-149 Acceptor1 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=88, name=qtp1396567385-88 Acceptor0 SelectChannelConnector@0.0.0.0:57148, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=32, name=qtp137322702-32 Acceptor0 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=213, name=qtp221024277-213 Acceptor3 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=101, name=qtp1455364682-101 Acceptor3 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=135, name=qtp435252846-135 Acceptor1 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=200, name=qtp1255434575-200 Acceptor0 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=57, name=qtp1164548552-57 Acceptor0 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=89, name=qtp1396567385-89 Acceptor1 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=116, name=qtp1594811832-116 Acceptor2 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=201, name=qtp1255434575-201 Acceptor1 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=247, name=qtp1571259428-247 Acceptor3 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=211, name=qtp221024277-211 Acceptor1 SelectChannelConnector@0.0.0.0:60070, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=203, name=qtp1255434575-203 Acceptor3 SelectChannelConnector@0.0.0.0:13276, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=114, name=qtp1594811832-114 Acceptor0 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=274, name=qtp955386212-274 Acceptor3 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=192, name=qtp1114740402-192 Acceptor3 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=90, name=qtp1396567385-90 Acceptor2 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=273, name=qtp955386212-273 Acceptor2 SelectChannelConnector@0.0.0.0:21766, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=271, name=qtp955386212-271 Acceptor0 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=189, name=qtp1114740402-189 Acceptor0 SelectChannelConnector@0.0.0.0:38544, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=151, name=qtp1566364831-151 Acceptor3 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=100, name=qtp1455364682-100 Acceptor2 SelectChannelConnector@0.0.0.0:38328, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  37) Thread[id=33, name=qtp137322702-33 Acceptor1 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  38) Thread[id=210, name=qtp221024277-210 Acceptor0 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  39) Thread[id=73, name=qtp1342965186-73 Acceptor2 SelectChannelConnector@0.0.0.0:55023, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  40) Thread[id=136, name=qtp435252846-136 Acceptor2 SelectChannelConnector@0.0.0.0:56846, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  41) Thread[id=134, name=qtp435252846-134 Acceptor0 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  42) Thread[id=290, name=qtp968634387-290 Acceptor1 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  43) Thread[id=202, name=qtp1255434575-202 Acceptor2 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  44) Thread[id=117, name=qtp1594811832-117 Acceptor3 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  45) Thread[id=35, name=qtp137322702-35 Acceptor3 SelectChannelConnector@0.0.0.0:17417, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  46) Thread[id=292, name=qtp968634387-292 Acceptor3 SelectChannelConnector@0.0.0.0:34474, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  47) Thread[id=34, name=qtp137322702-34 Acceptor2 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  48) Thread[id=230, name=qtp1500381475-230 Acceptor0 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  49) Thread[id=233, name=qtp1500381475-233 Acceptor3 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  50) Thread[id=212, name=qtp221024277-212 Acceptor2 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  51) Thread[id=244, name=qtp1571259428-244 Acceptor0 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  52) Thread[id=291, name=qtp968634387-291 Acceptor2 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  53) Thread[id=232, name=qtp1500381475-232 Acceptor2 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  54) Thread[id=49, name=qtp1220869853-49 Acceptor3 SelectChannelConnector@0.0.0.0:28190, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  55) Thread[id=74, name=qtp1342965186-74 Acceptor3 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  56) Thread[id=137, name=qtp435252846-137 Acceptor3 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  57) Thread[id=71, name=qtp1342965186-71 Acceptor0 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  58) Thread[id=46, name=qtp1220869853-46 Acceptor0 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  59) Thread[id=231, name=qtp1500381475-231 Acceptor1 SelectChannelConnector@0.0.0.0:30272, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  60) Thread[id=190, name=qtp1114740402-190 Acceptor1 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  61) Thread[id=98, name=qtp1455364682-98 Acceptor0 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  62) Thread[id=150, name=qtp1566364831-150 Acceptor2 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  63) Thread[id=60, name=qtp1164548552-60 Acceptor3 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  64) Thread[id=72, name=qtp1342965186-72 Acceptor1 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([6FB52D701538DE0]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=289, name=qtp968634387-289 Acceptor0 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=99, name=qtp1455364682-99 Acceptor1 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=59, name=qtp1164548552-59 Acceptor2 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=245, name=qtp1571259428-245 Acceptor1 SelectChannelConnector@0.0.0.0:63180, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=272, name=qtp955386212-272 Acceptor1 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=48, name=qtp1220869853-48 Acceptor2 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=148, name=qtp1566364831-148 Acceptor0 SelectChannelConnector@0.0.0.0:17939, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=58, name=qtp1164548552-58 Acceptor1 SelectChannelConnector@0.0.0.0:57149, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=91, name=qtp1396567385-91 Acceptor3 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=47, name=qtp1220869853-47 Acceptor1 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=191, name=qtp1114740402-191 Acceptor2 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=115, name=qtp1594811832-115 Acceptor1 SelectChannelConnector@0.0.0.0:20339, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=149, name=qtp1566364831-149 Acceptor1 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=246, name=qtp1571259428-246 Acceptor2 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=88, name=qtp1396567385-88 Acceptor0 SelectChannelConnector@0.0.0.0:57148, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=32, name=qtp137322702-32 Acceptor0 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=213, name=qtp221024277-213 Acceptor3 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=101, name=qtp1455364682-101 Acceptor3 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=135, name=qtp435252846-135 Acceptor1 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=200, name=qtp1255434575-200 Acceptor0 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   21) Thread[id=57, name=qtp1164548552-57 Acceptor0 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   22) Thread[id=89, name=qtp1396567385-89 Acceptor1 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   23) Thread[id=116, name=qtp1594811832-116 Acceptor2 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   24) Thread[id=201, name=qtp1255434575-201 Acceptor1 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   25) Thread[id=247, name=qtp1571259428-247 Acceptor3 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   26) Thread[id=211, name=qtp221024277-211 Acceptor1 SelectChannelConnector@0.0.0.0:60070, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   27) Thread[id=114, name=qtp1594811832-114 Acceptor0 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   28) Thread[id=203, name=qtp1255434575-203 Acceptor3 SelectChannelConnector@0.0.0.0:13276, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   29) Thread[id=274, name=qtp955386212-274 Acceptor3 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   30) Thread[id=192, name=qtp1114740402-192 Acceptor3 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   31) Thread[id=90, name=qtp1396567385-90 Acceptor2 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   32) Thread[id=273, name=qtp955386212-273 Acceptor2 SelectChannelConnector@0.0.0.0:21766, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   33) Thread[id=271, name=qtp955386212-271 Acceptor0 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   34) Thread[id=33, name=qtp137322702-33 Acceptor1 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   35) Thread[id=100, name=qtp1455364682-100 Acceptor2 SelectChannelConnector@0.0.0.0:38328, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   36) Thread[id=151, name=qtp1566364831-151 Acceptor3 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   37) Thread[id=189, name=qtp1114740402-189 Acceptor0 SelectChannelConnector@0.0.0.0:38544, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   38) Thread[id=73, name=qtp1342965186-73 Acceptor2 SelectChannelConnector@0.0.0.0:55023, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   39) Thread[id=210, name=qtp221024277-210 Acceptor0 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   40) Thread[id=134, name=qtp435252846-134 Acceptor0 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   41) Thread[id=136, name=qtp435252846-136 Acceptor2 SelectChannelConnector@0.0.0.0:56846, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   42) Thread[id=290, name=qtp968634387-290 Acceptor1 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   43) Thread[id=202, name=qtp1255434575-202 Acceptor2 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   44) Thread[id=117, name=qtp1594811832-117 Acceptor3 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   45) Thread[id=35, name=qtp137322702-35 Acceptor3 SelectChannelConnector@0.0.0.0:17417, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   46) Thread[id=34, name=qtp137322702-34 Acceptor2 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   47) Thread[id=292, name=qtp968634387-292 Acceptor3 SelectChannelConnector@0.0.0.0:34474, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   48) Thread[id=230, name=qtp1500381475-230 Acceptor0 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   49) Thread[id=233, name=qtp1500381475-233 Acceptor3 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   50) Thread[id=212, name=qtp221024277-212 Acceptor2 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   51) Thread[id=244, name=qtp1571259428-244 Acceptor0 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   52) Thread[id=291, name=qtp968634387-291 Acceptor2 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   53) Thread[id=232, name=qtp1500381475-232 Acceptor2 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   54) Thread[id=49, name=qtp1220869853-49 Acceptor3 SelectChannelConnector@0.0.0.0:28190, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   55) Thread[id=74, name=qtp1342965186-74 Acceptor3 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   56) Thread[id=137, name=qtp435252846-137 Acceptor3 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   57) Thread[id=71, name=qtp1342965186-71 Acceptor0 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   58) Thread[id=46, name=qtp1220869853-46 Acceptor0 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   59) Thread[id=231, name=qtp1500381475-231 Acceptor1 SelectChannelConnector@0.0.0.0:30272, state=RUNNABLE, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   60) Thread[id=190, name=qtp1114740402-190 Acceptor1 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   61) Thread[id=98, name=qtp1455364682-98 Acceptor0 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   62) Thread[id=150, name=qtp1566364831-150 Acceptor2 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   63) Thread[id=60, name=qtp1164548552-60 Acceptor3 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   64) Thread[id=72, name=qtp1342965186-72 Acceptor1 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=289, name=qtp968634387-289 Acceptor0 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=99, name=qtp1455364682-99 Acceptor1 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=59, name=qtp1164548552-59 Acceptor2 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=245, name=qtp1571259428-245 Acceptor1 SelectChannelConnector@0.0.0.0:63180, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=272, name=qtp955386212-272 Acceptor1 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=48, name=qtp1220869853-48 Acceptor2 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=148, name=qtp1566364831-148 Acceptor0 SelectChannelConnector@0.0.0.0:17939, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=58, name=qtp1164548552-58 Acceptor1 SelectChannelConnector@0.0.0.0:57149, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=91, name=qtp1396567385-91 Acceptor3 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=47, name=qtp1220869853-47 Acceptor1 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=191, name=qtp1114740402-191 Acceptor2 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=115, name=qtp1594811832-115 Acceptor1 SelectChannelConnector@0.0.0.0:20339, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=149, name=qtp1566364831-149 Acceptor1 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=246, name=qtp1571259428-246 Acceptor2 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=88, name=qtp1396567385-88 Acceptor0 SelectChannelConnector@0.0.0.0:57148, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=32, name=qtp137322702-32 Acceptor0 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=213, name=qtp221024277-213 Acceptor3 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=101, name=qtp1455364682-101 Acceptor3 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=135, name=qtp435252846-135 Acceptor1 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=200, name=qtp1255434575-200 Acceptor0 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  21) Thread[id=57, name=qtp1164548552-57 Acceptor0 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  22) Thread[id=89, name=qtp1396567385-89 Acceptor1 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  23) Thread[id=116, name=qtp1594811832-116 Acceptor2 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  24) Thread[id=201, name=qtp1255434575-201 Acceptor1 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  25) Thread[id=247, name=qtp1571259428-247 Acceptor3 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  26) Thread[id=211, name=qtp221024277-211 Acceptor1 SelectChannelConnector@0.0.0.0:60070, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  27) Thread[id=114, name=qtp1594811832-114 Acceptor0 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  28) Thread[id=203, name=qtp1255434575-203 Acceptor3 SelectChannelConnector@0.0.0.0:13276, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  29) Thread[id=274, name=qtp955386212-274 Acceptor3 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  30) Thread[id=192, name=qtp1114740402-192 Acceptor3 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  31) Thread[id=90, name=qtp1396567385-90 Acceptor2 SelectChannelConnector@0.0.0.0:57148, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  32) Thread[id=273, name=qtp955386212-273 Acceptor2 SelectChannelConnector@0.0.0.0:21766, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  33) Thread[id=271, name=qtp955386212-271 Acceptor0 SelectChannelConnector@0.0.0.0:21766, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  34) Thread[id=33, name=qtp137322702-33 Acceptor1 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  35) Thread[id=100, name=qtp1455364682-100 Acceptor2 SelectChannelConnector@0.0.0.0:38328, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  36) Thread[id=151, name=qtp1566364831-151 Acceptor3 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  37) Thread[id=189, name=qtp1114740402-189 Acceptor0 SelectChannelConnector@0.0.0.0:38544, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  38) Thread[id=73, name=qtp1342965186-73 Acceptor2 SelectChannelConnector@0.0.0.0:55023, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  39) Thread[id=210, name=qtp221024277-210 Acceptor0 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  40) Thread[id=134, name=qtp435252846-134 Acceptor0 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  41) Thread[id=136, name=qtp435252846-136 Acceptor2 SelectChannelConnector@0.0.0.0:56846, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  42) Thread[id=290, name=qtp968634387-290 Acceptor1 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  43) Thread[id=202, name=qtp1255434575-202 Acceptor2 SelectChannelConnector@0.0.0.0:13276, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  44) Thread[id=117, name=qtp1594811832-117 Acceptor3 SelectChannelConnector@0.0.0.0:20339, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  45) Thread[id=35, name=qtp137322702-35 Acceptor3 SelectChannelConnector@0.0.0.0:17417, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  46) Thread[id=34, name=qtp137322702-34 Acceptor2 SelectChannelConnector@0.0.0.0:17417, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  47) Thread[id=292, name=qtp968634387-292 Acceptor3 SelectChannelConnector@0.0.0.0:34474, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  48) Thread[id=230, name=qtp1500381475-230 Acceptor0 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  49) Thread[id=233, name=qtp1500381475-233 Acceptor3 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  50) Thread[id=212, name=qtp221024277-212 Acceptor2 SelectChannelConnector@0.0.0.0:60070, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  51) Thread[id=244, name=qtp1571259428-244 Acceptor0 SelectChannelConnector@0.0.0.0:63180, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  52) Thread[id=291, name=qtp968634387-291 Acceptor2 SelectChannelConnector@0.0.0.0:34474, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  53) Thread[id=232, name=qtp1500381475-232 Acceptor2 SelectChannelConnector@0.0.0.0:30272, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  54) Thread[id=49, name=qtp1220869853-49 Acceptor3 SelectChannelConnector@0.0.0.0:28190, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  55) Thread[id=74, name=qtp1342965186-74 Acceptor3 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  56) Thread[id=137, name=qtp435252846-137 Acceptor3 SelectChannelConnector@0.0.0.0:56846, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  57) Thread[id=71, name=qtp1342965186-71 Acceptor0 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  58) Thread[id=46, name=qtp1220869853-46 Acceptor0 SelectChannelConnector@0.0.0.0:28190, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  59) Thread[id=231, name=qtp1500381475-231 Acceptor1 SelectChannelConnector@0.0.0.0:30272, state=RUNNABLE, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  60) Thread[id=190, name=qtp1114740402-190 Acceptor1 SelectChannelConnector@0.0.0.0:38544, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  61) Thread[id=98, name=qtp1455364682-98 Acceptor0 SelectChannelConnector@0.0.0.0:38328, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  62) Thread[id=150, name=qtp1566364831-150 Acceptor2 SelectChannelConnector@0.0.0.0:17939, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  63) Thread[id=60, name=qtp1164548552-60 Acceptor3 SelectChannelConnector@0.0.0.0:57149, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  64) Thread[id=72, name=qtp1342965186-72 Acceptor1 SelectChannelConnector@0.0.0.0:55023, state=BLOCKED, group=TGRP-TestReplicationHandler]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([6FB52D701538DE0]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.CacheHeaderTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest:     1) Thread[id=176, name=qtp841817769-176 Acceptor2 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=177, name=qtp841817769-177 Acceptor3 SelectChannelConnector@0.0.0.0:64937, state=RUNNABLE, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=175, name=qtp841817769-175 Acceptor1 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=174, name=qtp841817769-174 Acceptor0 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest: 
   1) Thread[id=176, name=qtp841817769-176 Acceptor2 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=177, name=qtp841817769-177 Acceptor3 SelectChannelConnector@0.0.0.0:64937, state=RUNNABLE, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=175, name=qtp841817769-175 Acceptor1 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=174, name=qtp841817769-174 Acceptor0 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([6FB52D701538DE0]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.servlet.CacheHeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=176, name=qtp841817769-176 Acceptor2 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=177, name=qtp841817769-177 Acceptor3 SelectChannelConnector@0.0.0.0:64937, state=RUNNABLE, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=175, name=qtp841817769-175 Acceptor1 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=174, name=qtp841817769-174 Acceptor0 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=176, name=qtp841817769-176 Acceptor2 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=177, name=qtp841817769-177 Acceptor3 SelectChannelConnector@0.0.0.0:64937, state=RUNNABLE, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=175, name=qtp841817769-175 Acceptor1 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=174, name=qtp841817769-174 Acceptor0 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([6FB52D701538DE0]:0)




Build Log:
[...truncated 7705 lines...]
[junit4:junit4] Suite: org.apache.solr.handler.TestReplicationHandler
[junit4:junit4]   2> 408 T25 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 669 T25 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:17417
[junit4:junit4]   2> 698 T25 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 698 T25 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master
[junit4:junit4]   2> 699 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/'
[junit4:junit4]   2> 728 T25 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 729 T25 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 729 T25 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master
[junit4:junit4]   2> 730 T25 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/solr.xml
[junit4:junit4]   2> 730 T25 oasc.CoreContainer.<init> New CoreContainer 630290952
[junit4:junit4]   2> 731 T25 oasc.CoreContainer$Initializer.initialize no solr.xml file found - using default
[junit4:junit4]   2> 731 T25 oasc.CoreContainer.load Loading CoreContainer using Solr Home: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/'
[junit4:junit4]   2> 731 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/'
[junit4:junit4]   2> 763 T25 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 806 T25 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/collection1
[junit4:junit4]   2> 807 T25 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/collection1/'
[junit4:junit4]   2> 841 T25 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 890 T25 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 891 T25 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 894 T25 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 913 T25 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 914 T25 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/collection1/, dataDir=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/collection1/data/
[junit4:junit4]   2> 915 T25 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 934 T25 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/collection1/data/index/
[junit4:junit4]   2> 935 T25 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/collection1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 936 T25 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/collection1/data/index forceNew:false
[junit4:junit4]   2> 946 T25 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.handler.TestReplicationHandler$SolrInstance-1351571251172/master/collection1/data/index,segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 946 T25 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 947 T25 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 975 T25 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 975 T25 oasc.RequestHandlers.initHandlersFromConfig 

[...truncated too long message...]

 at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=174, name=qtp841817769-174 Acceptor0 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=es_ES, timezone=Asia/Aqtau
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=29394808,total=95289344
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestJmxMonitoredMap, TestFaceting, TestCoreContainer, LoggingHandlerTest, AlternateDirectoryTest, TestReload, TestIBSimilarityFactory, TestRTGBase, PathHierarchyTokenizerFactoryTest, TestBinaryResponseWriter, TestSystemIdResolver, TestJoin, JSONWriterTest, TestFastWriter, CacheHeaderTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CacheHeaderTest -Dtests.seed=6FB52D701538DE0 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_ES -Dtests.timezone=Asia/Aqtau -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR   0.00s J0 | CacheHeaderTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.servlet.CacheHeaderTest: 
[junit4:junit4]    >    1) Thread[id=176, name=qtp841817769-176 Acceptor2 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=177, name=qtp841817769-177 Acceptor3 SelectChannelConnector@0.0.0.0:64937, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=175, name=qtp841817769-175 Acceptor1 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=174, name=qtp841817769-174 Acceptor0 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([6FB52D701538DE0]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=176, name=qtp841817769-176 Acceptor2 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=177, name=qtp841817769-177 Acceptor3 SelectChannelConnector@0.0.0.0:64937, state=RUNNABLE, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=175, name=qtp841817769-175 Acceptor1 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=174, name=qtp841817769-174 Acceptor0 SelectChannelConnector@0.0.0.0:64937, state=BLOCKED, group=TGRP-CacheHeaderTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([6FB52D701538DE0]:0)
[junit4:junit4] Completed on J0 in 26.39s, 5 tests, 2 errors <<< FAILURES!

[...truncated 140 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1159: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 845 ignored

Total time: 19 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 819 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/819/

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.request.TestRemoteStreaming

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming:     1) Thread[id=39, name=qtp751294816-39 Acceptor2 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=37, name=qtp751294816-37 Acceptor0 SelectChannelConnector@0.0.0.0:59895, state=RUNNABLE, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=40, name=qtp751294816-40 Acceptor3 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=38, name=qtp751294816-38 Acceptor1 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming: 
   1) Thread[id=39, name=qtp751294816-39 Acceptor2 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=37, name=qtp751294816-37 Acceptor0 SelectChannelConnector@0.0.0.0:59895, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=40, name=qtp751294816-40 Acceptor3 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=38, name=qtp751294816-38 Acceptor1 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([C51BE694BBAD5B5B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.request.TestRemoteStreaming

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=39, name=qtp751294816-39 Acceptor2 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=37, name=qtp751294816-37 Acceptor0 SelectChannelConnector@0.0.0.0:59895, state=RUNNABLE, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=40, name=qtp751294816-40 Acceptor3 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=38, name=qtp751294816-38 Acceptor1 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=39, name=qtp751294816-39 Acceptor2 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=37, name=qtp751294816-37 Acceptor0 SelectChannelConnector@0.0.0.0:59895, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=40, name=qtp751294816-40 Acceptor3 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=38, name=qtp751294816-38 Acceptor1 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([C51BE694BBAD5B5B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.SolrCmdDistributorTest

Error Message:
20 threads leaked from SUITE scope at org.apache.solr.update.SolrCmdDistributorTest:     1) Thread[id=106, name=qtp680387598-106 Acceptor3 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=91, name=qtp813186733-91 Acceptor1 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=145, name=qtp170949019-145 Acceptor0 SelectChannelConnector@0.0.0.0:30981, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=103, name=qtp680387598-103 Acceptor0 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=134, name=qtp779119119-134 Acceptor3 SelectChannelConnector@0.0.0.0:30980, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=117, name=qtp829021858-117 Acceptor0 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=133, name=qtp779119119-133 Acceptor2 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=132, name=qtp779119119-132 Acceptor1 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=93, name=qtp813186733-93 Acceptor3 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=90, name=qtp813186733-90 Acceptor0 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=120, name=qtp829021858-120 Acceptor3 SelectChannelConnector@0.0.0.0:30979, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=119, name=qtp829021858-119 Acceptor2 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=131, name=qtp779119119-131 Acceptor0 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=92, name=qtp813186733-92 Acceptor2 SelectChannelConnector@0.0.0.0:30973, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=147, name=qtp170949019-147 Acceptor2 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=104, name=qtp680387598-104 Acceptor1 SelectChannelConnector@0.0.0.0:30978, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=118, name=qtp829021858-118 Acceptor1 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=105, name=qtp680387598-105 Acceptor2 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=146, name=qtp170949019-146 Acceptor1 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=148, name=qtp170949019-148 Acceptor3 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 20 threads leaked from SUITE scope at org.apache.solr.update.SolrCmdDistributorTest: 
   1) Thread[id=106, name=qtp680387598-106 Acceptor3 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=91, name=qtp813186733-91 Acceptor1 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=145, name=qtp170949019-145 Acceptor0 SelectChannelConnector@0.0.0.0:30981, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=103, name=qtp680387598-103 Acceptor0 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=134, name=qtp779119119-134 Acceptor3 SelectChannelConnector@0.0.0.0:30980, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=117, name=qtp829021858-117 Acceptor0 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=133, name=qtp779119119-133 Acceptor2 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=132, name=qtp779119119-132 Acceptor1 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=93, name=qtp813186733-93 Acceptor3 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=90, name=qtp813186733-90 Acceptor0 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=120, name=qtp829021858-120 Acceptor3 SelectChannelConnector@0.0.0.0:30979, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=119, name=qtp829021858-119 Acceptor2 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=131, name=qtp779119119-131 Acceptor0 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=92, name=qtp813186733-92 Acceptor2 SelectChannelConnector@0.0.0.0:30973, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=147, name=qtp170949019-147 Acceptor2 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=104, name=qtp680387598-104 Acceptor1 SelectChannelConnector@0.0.0.0:30978, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=118, name=qtp829021858-118 Acceptor1 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=105, name=qtp680387598-105 Acceptor2 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=146, name=qtp170949019-146 Acceptor1 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=148, name=qtp170949019-148 Acceptor3 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([C51BE694BBAD5B5B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.update.SolrCmdDistributorTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=91, name=qtp813186733-91 Acceptor1 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    2) Thread[id=106, name=qtp680387598-106 Acceptor3 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    3) Thread[id=103, name=qtp680387598-103 Acceptor0 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    4) Thread[id=145, name=qtp170949019-145 Acceptor0 SelectChannelConnector@0.0.0.0:30981, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    5) Thread[id=134, name=qtp779119119-134 Acceptor3 SelectChannelConnector@0.0.0.0:30980, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    6) Thread[id=117, name=qtp829021858-117 Acceptor0 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    7) Thread[id=133, name=qtp779119119-133 Acceptor2 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    8) Thread[id=132, name=qtp779119119-132 Acceptor1 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)    9) Thread[id=90, name=qtp813186733-90 Acceptor0 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   10) Thread[id=93, name=qtp813186733-93 Acceptor3 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   11) Thread[id=120, name=qtp829021858-120 Acceptor3 SelectChannelConnector@0.0.0.0:30979, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   12) Thread[id=131, name=qtp779119119-131 Acceptor0 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   13) Thread[id=119, name=qtp829021858-119 Acceptor2 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   14) Thread[id=92, name=qtp813186733-92 Acceptor2 SelectChannelConnector@0.0.0.0:30973, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   15) Thread[id=147, name=qtp170949019-147 Acceptor2 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   16) Thread[id=104, name=qtp680387598-104 Acceptor1 SelectChannelConnector@0.0.0.0:30978, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   17) Thread[id=118, name=qtp829021858-118 Acceptor1 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   18) Thread[id=105, name=qtp680387598-105 Acceptor2 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   19) Thread[id=146, name=qtp170949019-146 Acceptor1 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)   20) Thread[id=148, name=qtp170949019-148 Acceptor3 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)         at java.lang.Thread.run(Thread.java:679)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=91, name=qtp813186733-91 Acceptor1 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   2) Thread[id=106, name=qtp680387598-106 Acceptor3 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   3) Thread[id=103, name=qtp680387598-103 Acceptor0 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   4) Thread[id=145, name=qtp170949019-145 Acceptor0 SelectChannelConnector@0.0.0.0:30981, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   5) Thread[id=134, name=qtp779119119-134 Acceptor3 SelectChannelConnector@0.0.0.0:30980, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   6) Thread[id=117, name=qtp829021858-117 Acceptor0 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   7) Thread[id=133, name=qtp779119119-133 Acceptor2 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   8) Thread[id=132, name=qtp779119119-132 Acceptor1 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
   9) Thread[id=90, name=qtp813186733-90 Acceptor0 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  10) Thread[id=93, name=qtp813186733-93 Acceptor3 SelectChannelConnector@0.0.0.0:30973, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  11) Thread[id=120, name=qtp829021858-120 Acceptor3 SelectChannelConnector@0.0.0.0:30979, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  12) Thread[id=131, name=qtp779119119-131 Acceptor0 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  13) Thread[id=119, name=qtp829021858-119 Acceptor2 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  14) Thread[id=92, name=qtp813186733-92 Acceptor2 SelectChannelConnector@0.0.0.0:30973, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  15) Thread[id=147, name=qtp170949019-147 Acceptor2 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  16) Thread[id=104, name=qtp680387598-104 Acceptor1 SelectChannelConnector@0.0.0.0:30978, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  17) Thread[id=118, name=qtp829021858-118 Acceptor1 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  18) Thread[id=105, name=qtp680387598-105 Acceptor2 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  19) Thread[id=146, name=qtp170949019-146 Acceptor1 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
  20) Thread[id=148, name=qtp170949019-148 Acceptor3 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:679)
	at __randomizedtesting.SeedInfo.seed([C51BE694BBAD5B5B]:0)




Build Log:
[...truncated 7693 lines...]
[junit4:junit4] Suite: org.apache.solr.request.TestRemoteStreaming
[junit4:junit4]   2> 0 T30 oas.SolrTestCaseJ4.initCore ####initCore
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-TestRemoteStreaming-1351539950307
[junit4:junit4]   2> 0 T30 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4:junit4]   2> 409 T30 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 670 T30 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:59895
[junit4:junit4]   2> 698 T30 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 699 T30 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: solr/
[junit4:junit4]   2> 699 T30 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: 'solr/'
[junit4:junit4]   2> 737 T30 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 737 T30 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 738 T30 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: solr/
[junit4:junit4]   2> 738 T30 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solr/solr.xml
[junit4:junit4]   2> 739 T30 oasc.CoreContainer.<init> New CoreContainer 1252365537
[junit4:junit4]   2> 739 T30 oasc.CoreContainer$Initializer.initialize no solr.xml file found - using default
[junit4:junit4]   2> 740 T30 oasc.CoreContainer.load Loading CoreContainer using Solr Home: 'solr/'
[junit4:junit4]   2> 740 T30 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/'
[junit4:junit4]   2> 773 T30 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 821 T30 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: solr/collection1
[junit4:junit4]   2> 822 T30 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: 'solr/collection1/'
[junit4:junit4]   2> 945 T30 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1026 T30 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1029 T30 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1062 T30 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1730 T30 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1741 T30 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1745 T30 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1781 T30 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1799 T30 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1804 T30 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1809 T30 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1812 T30 oasc.SolrCore.<init> [collection1] Opening new SolrCore at solr/collection1/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-TestRemoteStreaming-1351539950307/
[junit4:junit4]   2> 1812 T30 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4b9a0203
[junit4:junit4]   2> 1814 T30 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-TestRemoteStreaming-1351539950307/index/
[junit4:junit4]   2> 1815 T30 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-TestRemoteStreaming-1351539950307/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1819 T30 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/solrtest-TestRemoteStreaming-1351539950307/index forceNew:false
[junit4:junit4]   2> 1822 T30 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1822 T30 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1823 T30 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
[junit4:junit4]   2> 1825 T30 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1825 T30 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
[junit4:junit4]   2> 1826 T30 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1826 T30 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
[junit4:junit4]   2> 1828 T30 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1828 T30 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1829 T30 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1830 T30 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1831 T30 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1832 T30 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
[junit4:junit4]   2> 1833 T30 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
[junit4:junit4]   2> 1834 T30 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1835 T30 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
[junit4:junit4]   2> 1835 T30 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
[junit4:junit4]   2> 1836 T30 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
[junit4:junit4]   2> 1837 T30 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1838 T30 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1839 T30 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1840 T30 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1841 T30 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1842 T30 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1843 T30 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1843 T30 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
[junit4:junit4]   2> 1844 T30 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
[junit4:junit4]   2> 1845 T30 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
[junit4:junit4]   2> 1853 T30 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1859 T30 oasc.SolrCore.initDeprecatedSupport WARNING solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
[junit4:junit4]   2> 1861 T30 oasc.SolrCore.initDeprecatedSupport WARNING adding ShowFileRequestHandler with hidden files: []
[junit4:junit4]   2> 1865 T30 oass.SolrIndexSearcher.<init> Opening Searcher@787db430 main
[junit4:junit4]   2> 1866 T30 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1867 T30 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1867 T30 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1868 T30 oashc.SpellCheckComponent.inform Initializing spell checkers
[junit4:junit4]   2> 1875 T30 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
[junit4:junit4]   2> 2139 T30 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2140 T30 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2141 T30 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2141 T30 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2142 T30 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2142 T30 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2143 T30 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2143 T30 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2144 T30 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2144 T30 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2170 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@787db430 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2179 T30 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2179 T30 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4:junit4]   2> 2180 T30 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2180 T30 oas.SolrJettyTestBase.createJetty Jetty Assigned Port#59895
[junit4:junit4]   2> 2189 T30 oas.SolrTestCaseJ4.setUp ###Starting testMakeDeleteAllUrl
[junit4:junit4]   2> 2262 T30 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@2cc47220
[junit4:junit4]   2> 3545 T44 C4 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3546 T44 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3548 T44 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1234]} 0 30
[junit4:junit4]   2> 3708 T43 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 3711 T43 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_2,generation=2,filenames=[_0.tis, _0.nrm, _0.fnm, _0.tii, _0.frq, segments_2, _0.fdx, _0.si, _0.prx, _0.fdt]
[junit4:junit4]   2> 3711 T43 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 3713 T43 C4 oass.SolrIndexSearcher.<init> Opening Searcher@6112c9f main
[junit4:junit4]   2> 3713 T43 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3714 T43 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 3716 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6112c9f main{StandardDirectoryReader(segments_2:3 _0(4.1):C1)}
[junit4:junit4]   2> 3716 T43 C4 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 8
[junit4:junit4]   2> 3797 T42 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=1 status=0 QTime=57 
[junit4:junit4]   2> 3810 T44 C4 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
[junit4:junit4]   2> 3811 T44 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 3812 T44 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_2,generation=2,filenames=[_0.tis, _0.nrm, _0.fnm, _0.tii, _0.frq, segments_2, _0.fdx, _0.si, _0.prx, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 3812 T44 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 3813 T44 C4 oass.SolrIndexSearcher.<init> Opening Searcher@55991e21 main
[junit4:junit4]   2> 3814 T44 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3814 T44 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 3816 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@55991e21 main{StandardDirectoryReader(segments_3:4)}
[junit4:junit4]   2> 3816 T44 C4 UPDATE [collection1] webapp=/solr path=/update params={commit=true&stream.body=<delete><query>*:*</query></delete>} {deleteByQuery=*:*,commit=} 0 6
[junit4:junit4]   2> 3831 T43 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=0 status=0 QTime=1 
[junit4:junit4]   2> 3832 T30 oas.SolrTestCaseJ4.tearDown ###Ending testMakeDeleteAllUrl
[junit4:junit4]   2> 3839 T30 oas.SolrTestCaseJ4.setUp ###Starting testQtUpdateFails
[junit4:junit4]   2> 3845 T44 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1234]} 0 1
[junit4:junit4]   2> 3849 T42 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 3852 T42 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_4,generation=4,filenames=[_1.fdx, _1.prx, _1.tis, _1.frq, _1.fdt, _1.si, _1.tii, _1.fnm, _1.nrm, segments_4]
[junit4:junit4]   2> 3852 T42 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 3854 T42 C4 oass.SolrIndexSearcher.<init> Opening Searcher@79ccc6c8 main
[junit4:junit4]   2> 3855 T42 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3856 T42 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 3858 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79ccc6c8 main{StandardDirectoryReader(segments_4:6 _1(4.1):C1)}
[junit4:junit4]   2> 3859 T42 C4 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 11
[junit4:junit4]   2> 3863 T43 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 3867 T30 oas.SolrTestCaseJ4.tearDown ###Ending testQtUpdateFails
[junit4:junit4]   2> 3875 T30 oas.SolrTestCaseJ4.setUp ###Starting testNoUrlAccess
[junit4:junit4]   2> 3880 T42 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1234]} 0 2
[junit4:junit4]   2> 3884 T42 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 3888 T42 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_4,generation=4,filenames=[_1.fdx, _1.prx, _1.tis, _1.frq, _1.fdt, _1.si, _1.tii, _1.fnm, _1.nrm, segments_4]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_5,generation=5,filenames=[_2.si, _2.tis, _2.prx, _2.fdt, _2.frq, _2.tii, segments_5, _2.fdx, _2.nrm, _2.fnm]
[junit4:junit4]   2> 3888 T42 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 5
[junit4:junit4]   2> 3890 T42 C4 oass.SolrIndexSearcher.<init> Opening Searcher@277889e9 main
[junit4:junit4]   2> 3891 T42 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3892 T42 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 3894 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@277889e9 main{StandardDirectoryReader(segments_5:10 _2(4.1):C1)}
[junit4:junit4]   2> 3895 T42 C4 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 12
[junit4:junit4]   2> 3899 T43 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 3911 T44 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&stream.url=http://127.0.0.1:59895/solr/update?commit%3Dtrue%26stream.body%3D%253Cdelete%253E%253Cquery%253E*%253A*%253C%252Fquery%253E%253C%252Fdelete%253E&q=*:*&version=2} hits=1 status=0 QTime=6 
[junit4:junit4]   2> 3915 T42 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 3916 T30 oas.SolrTestCaseJ4.tearDown ###Ending testNoUrlAccess
[junit4:junit4]   2> 3926 T30 oas.SolrTestCaseJ4.setUp ###Starting testStreamUrl
[junit4:junit4]   2> 3931 T43 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1234]} 0 1
[junit4:junit4]   2> 3935 T44 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 3938 T44 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_5,generation=5,filenames=[_2.si, _2.tis, _2.prx, _2.fdt, _2.frq, _2.tii, segments_5, _2.fdx, _2.nrm, _2.fnm]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@7b26c885 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2647e53a; maxCacheMB=0.2978515625 maxMergeSizeMB=0.2607421875)),segFN=segments_6,generation=6,filenames=[_3.fdx, _3.nrm, _3.tii, _3.frq, _3.si, _3.fdt, _3.prx, _3.fnm, segments_6, _3.tis]
[junit4:junit4]   2> 3938 T44 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 6
[junit4:junit4]   2> 3940 T44 C4 oass.SolrIndexSearcher.<init> Opening Searcher@78e785aa main
[junit4:junit4]   2> 3940 T44 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3940 T44 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 3944 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@78e785aa main{StandardDirectoryReader(segments_6:14 _3(4.1):C1)}
[junit4:junit4]   2> 3945 T44 C4 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&wt=javabin&commit=true&softCommit=false&version=2} {commit=} 0 11
[junit4:junit4]   2> 3950 T42 C4 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=id:1234&version=2} hits=1 status=0 QTime=1 
[junit4:junit4]   2> 3955 T42 C4 REQ [collection1] webapp=/solr path=/select params={wt=csv&q=*:*&fl=id} hits=1 status=0 QTime=0 
[junit4:junit4]   2> 3983 T43 C4 REQ [collection1] webapp=/solr path=/debug/dump params={wt=xml&stream.url=http://127.0.0.1:59895/solr/select?q%3D*:*%26fl%3Did%26wt%3Dcsv} status=0 QTime=28 
[junit4:junit4]   2> 3984 T30 oas.SolrTestCaseJ4.tearDown ###Ending testStreamUrl
[junit4:junit4]   2> 4031 T30 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1252365537
[junit4:junit4]   2> 4031 T30 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@2cc47220
[junit4:junit4]   2> 4043 T30 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=5,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=4,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 4044 T30 oasc.SolrCore.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 4044 T30 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 4044 T30 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 4046 T30 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 4051 T30 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 4152 T30 oejut.QueuedThreadPool.doStop WARNING 4 threads could not be stopped
[junit4:junit4]   2> 4152 T30 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 4163 T29 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 24260 T29 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming: 
[junit4:junit4]   2> 	   1) Thread[id=39, name=qtp751294816-39 Acceptor2 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=37, name=qtp751294816-37 Acceptor0 SelectChannelConnector@0.0.0.0:59895, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=40, name=qtp751294816-40 Acceptor3 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=38, name=qtp751294816-38 Acceptor1 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 24261 T29 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2> 	   1) Thread[id=39, name=qtp751294816-39 Acceptor2 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	   2) Thread[id=37, name=qtp751294816-37 Acceptor0 SelectChannelConnector@0.0.0.0:59895, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	   3) Thread[id=40, name=qtp751294816-40 Acceptor3 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	   4) Thread[id=38, name=qtp751294816-38 Acceptor1 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 27305 T29 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2> 	   1) Thread[id=39, name=qtp751294816-39 Acceptor2 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   2) Thread[id=37, name=qtp751294816-37 Acceptor0 SelectChannelConnector@0.0.0.0:59895, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   3) Thread[id=40, name=qtp751294816-40 Acceptor3 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 	   4) Thread[id=38, name=qtp751294816-38 Acceptor1 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]   2> 	        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]   2> 	        at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene3x, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=et_EE, timezone=Asia/Gaza
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=5,free=125105864,total=201981952
[junit4:junit4]   2> NOTE: All tests run in this JVM: [ClusterStateTest, TestSort, TestCodecSupport, StandardRequestHandlerTest, SuggesterTSTTest, StatelessScriptUpdateProcessorFactoryTest, RequiredFieldsTest, TestRemoteStreaming]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRemoteStreaming -Dtests.seed=C51BE694BBAD5B5B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=et_EE -Dtests.timezone=Asia/Gaza -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] ERROR   0.00s J1 | TestRemoteStreaming (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.request.TestRemoteStreaming: 
[junit4:junit4]    >    1) Thread[id=39, name=qtp751294816-39 Acceptor2 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=37, name=qtp751294816-37 Acceptor0 SelectChannelConnector@0.0.0.0:59895, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=40, name=qtp751294816-40 Acceptor3 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=38, name=qtp751294816-38 Acceptor1 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([C51BE694BBAD5B5B]:0)
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=39, name=qtp751294816-39 Acceptor2 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    2) Thread[id=37, name=qtp751294816-37 Acceptor0 SelectChannelConnector@0.0.0.0:59895, state=RUNNABLE, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    3) Thread[id=40, name=qtp751294816-40 Acceptor3 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >    4) Thread[id=38, name=qtp751294816-38 Acceptor1 SelectChannelConnector@0.0.0.0:59895, state=BLOCKED, group=TGRP-TestRemoteStreaming]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([C51BE694BBAD5B5B]:0)
[junit4:junit4] Completed on J1 in 27.36s, 4 tests, 2 errors <<< FAILURES!

[...truncated 509 lines...]
[junit4:junit4] Suite: org.apache.solr.update.SolrCmdDistributorTest
[junit4:junit4]   2> 4 T83 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> 384 T83 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 671 T83 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:30973
[junit4:junit4]   2> 698 T83 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 699 T83 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 699 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 727 T83 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 727 T83 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 728 T83 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 728 T83 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 728 T83 oasc.CoreContainer.<init> New CoreContainer 1697191652
[junit4:junit4]   2> 729 T83 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 729 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 762 T83 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 804 T83 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 805 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 806 T83 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 807 T83 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 841 T83 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 880 T83 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 881 T83 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 915 T83 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1545 T83 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1564 T83 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1569 T83 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1590 T83 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1598 T83 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1603 T83 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1605 T83 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1606 T83 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/control/data/
[junit4:junit4]   2> 1606 T83 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 1608 T83 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/control/data/index/
[junit4:junit4]   2> 1609 T83 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.SolrCmdDistributorTest-1351540031477/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1610 T83 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/control/data/index forceNew:false
[junit4:junit4]   2> 1614 T83 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5793a045 lockFactory=org.apache.lucene.store.NativeFSLockFactory@397c8a21),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1615 T83 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 1617 T83 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1618 T83 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1618 T83 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1619 T83 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1627 T83 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1635 T83 oass.SolrIndexSearcher.<init> Opening Searcher@1604f448 main
[junit4:junit4]   2> 1635 T83 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 1666 T83 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1667 T83 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1668 T83 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 1669 T83 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 1669 T83 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 1670 T83 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 1670 T83 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 1671 T83 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 1671 T83 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 1672 T83 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 1672 T83 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 1673 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 1691 T97 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1604f448 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1695 T83 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 1696 T83 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 1696 T83 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 1773 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 1800 T83 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 1803 T83 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:30978
[junit4:junit4]   2> 1804 T83 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1805 T83 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1805 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1844 T83 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 1845 T83 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 1846 T83 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 1846 T83 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 1847 T83 oasc.CoreContainer.<init> New CoreContainer 436898820
[junit4:junit4]   2> 1848 T83 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1848 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 1879 T83 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 1901 T83 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 1902 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 1904 T83 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1905 T83 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 1949 T83 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 1987 T83 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 1988 T83 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2029 T83 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2514 T83 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2530 T83 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2535 T83 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2554 T83 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2564 T83 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2569 T83 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2571 T83 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2572 T83 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard0/data/
[junit4:junit4]   2> 2572 T83 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 2574 T83 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard0/data/index/
[junit4:junit4]   2> 2575 T83 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard0/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2576 T83 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard0/data/index forceNew:false
[junit4:junit4]   2> 2578 T83 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@309763a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5fed2117),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2578 T83 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 2579 T83 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2580 T83 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2580 T83 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2581 T83 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2585 T83 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2590 T83 oass.SolrIndexSearcher.<init> Opening Searcher@4c3cafba main
[junit4:junit4]   2> 2590 T83 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 2591 T83 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2592 T83 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2592 T83 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 2592 T83 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 2593 T83 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 2593 T83 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 2593 T83 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 2594 T83 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 2594 T83 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 2594 T83 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 2595 T83 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 2595 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 2606 T83 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 2606 T111 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c3cafba main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2606 T83 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 2607 T83 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 2607 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 2620 T83 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 2623 T83 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:30979
[junit4:junit4]   2> 2624 T83 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2625 T83 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2625 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2655 T83 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 2655 T83 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 2656 T83 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 2656 T83 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 2656 T83 oasc.CoreContainer.<init> New CoreContainer 1365789774
[junit4:junit4]   2> 2657 T83 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2657 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 2675 T83 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 2689 T83 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 2689 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 2690 T83 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 2691 T83 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 2727 T83 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 2760 T83 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 2761 T83 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 2793 T83 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 3303 T83 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 3317 T83 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 3320 T83 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 3336 T83 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3343 T83 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 3348 T83 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3349 T83 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 3350 T83 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard1/data/
[junit4:junit4]   2> 3350 T83 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 3352 T83 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard1/data/index/
[junit4:junit4]   2> 3353 T83 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard1/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 3354 T83 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard1/data/index forceNew:false
[junit4:junit4]   2> 3357 T83 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@d7b3d24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7abcaec7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 3358 T83 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 3359 T83 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 3360 T83 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 3360 T83 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 3361 T83 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 3369 T83 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 3376 T83 oass.SolrIndexSearcher.<init> Opening Searcher@2048d02f main
[junit4:junit4]   2> 3376 T83 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 3378 T83 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 3378 T83 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 3379 T83 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 3379 T83 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3380 T83 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 3380 T83 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3381 T83 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3381 T83 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3382 T83 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3383 T83 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3383 T83 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3384 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 3400 T125 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2048d02f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 3400 T83 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 3401 T83 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3402 T83 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3402 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3415 T83 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 3418 T83 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:30980
[junit4:junit4]   2> 3419 T83 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3419 T83 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 3420 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3453 T83 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 3454 T83 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3455 T83 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 3455 T83 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 3456 T83 oasc.CoreContainer.<init> New CoreContainer 1430139993
[junit4:junit4]   2> 3456 T83 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3457 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 3484 T83 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3500 T83 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 3500 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 3501 T83 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 3502 T83 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 3531 T83 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 3565 T83 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 3565 T83 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 3589 T83 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 4012 T83 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 4022 T83 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 4026 T83 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 4043 T83 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4049 T83 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4054 T83 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4055 T83 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4056 T83 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard2/data/
[junit4:junit4]   2> 4056 T83 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 4057 T83 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard2/data/index/
[junit4:junit4]   2> 4057 T83 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard2/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 4058 T83 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard2/data/index forceNew:false
[junit4:junit4]   2> 4061 T83 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@557ce599 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4fbd7fe9),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4061 T83 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4062 T83 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 4062 T83 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 4063 T83 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 4063 T83 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 4068 T83 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 4073 T83 oass.SolrIndexSearcher.<init> Opening Searcher@1ec4fb90 main
[junit4:junit4]   2> 4073 T83 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4074 T83 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 4074 T83 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 4075 T83 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 4075 T83 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4075 T83 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 4076 T83 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4076 T83 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4076 T83 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4076 T83 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4077 T83 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4077 T83 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4077 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 4088 T139 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ec4fb90 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 4088 T83 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 4088 T83 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 4089 T83 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 4089 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4097 T83 oejs.Server.doStart jetty-8.1.2.v20120308
[junit4:junit4]   2> 4099 T83 oejs.AbstractConnector.doStart Started SelectChannelConnector@0.0.0.0:30981
[junit4:junit4]   2> 4100 T83 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4100 T83 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 4101 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for deduced Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 4122 T83 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4123 T83 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4123 T83 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr
[junit4:junit4]   2> 4124 T83 oasc.CoreContainer$Initializer.initialize looking for solr.xml: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/solr.xml
[junit4:junit4]   2> 4124 T83 oasc.CoreContainer.<init> New CoreContainer 890682541
[junit4:junit4]   2> 4124 T83 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 4125 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/'
[junit4:junit4]   2> 4145 T83 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4162 T83 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1
[junit4:junit4]   2> 4163 T83 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/'
[junit4:junit4]   2> 4164 T83 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 4165 T83 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/lib/README' to classloader
[junit4:junit4]   2> 4206 T83 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_41
[junit4:junit4]   2> 4239 T83 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig-tlog.xml
[junit4:junit4]   2> 4240 T83 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 4259 T83 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 4670 T83 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 4679 T83 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 4682 T83 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 4693 T83 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4698 T83 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 4701 T83 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4702 T83 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 4703 T83 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/, dataDir=./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard3/data/
[junit4:junit4]   2> 4703 T83 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4:junit4]   2> 4704 T83 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard3/data/index/
[junit4:junit4]   2> 4705 T83 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard3/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 4705 T83 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0/./org.apache.solr.update.SolrCmdDistributorTest-1351540031477/shard3/data/index forceNew:false
[junit4:junit4]   2> 4708 T83 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@29e012bb lockFactory=org.apache.lucene.store.NativeFSLockFactory@e04184a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 4708 T83 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 4709 T83 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 4709 T83 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 4710 T83 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 4710 T83 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 4715 T83 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 4719 T83 oass.SolrIndexSearcher.<init> Opening Searcher@4818cb9e main
[junit4:junit4]   2> 4719 T83 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 4720 T83 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 4720 T83 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 4721 T83 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
[junit4:junit4]   2> 4721 T83 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4721 T83 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
[junit4:junit4]   2> 4722 T83 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4722 T83 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4722 T83 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4722 T83 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4723 T83 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4723 T83 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4723 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=0&connTimeout=0&retry=false
[junit4:junit4]   2> 4733 T153 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4818cb9e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 4733 T83 oasc.CoreContainer.register registering core: collection1
[junit4:junit4]   2> 4734 T83 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J0
[junit4:junit4]   2> 4734 T83 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 4734 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@4acd2ea5
[junit4:junit4]   2> 5663 T96 C2 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5793a045 lockFactory=org.apache.lucene.store.NativeFSLockFactory@397c8a21),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5664 T96 C2 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5694 T96 C2 oass.SolrIndexSearcher.<init> Opening Searcher@34450bd5 realtime
[junit4:junit4]   2> 5694 T96 C2 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5697 T96 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1417192445956849664)} 0 81
[junit4:junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@27871523
[junit4:junit4]   2> 5827 T110 C3 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@309763a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5fed2117),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5828 T110 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5829 T110 C3 oass.SolrIndexSearcher.<init> Opening Searcher@7665a560 realtime
[junit4:junit4]   2> 5830 T110 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 5830 T110 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1417192446150836224)} 0 5
[junit4:junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@4dd25f53
[junit4:junit4]   2> 5837 T123 C4 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@d7b3d24 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7abcaec7),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5837 T123 C4 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5838 T123 C4 oass.SolrIndexSearcher.<init> Opening Searcher@536a887c realtime
[junit4:junit4]   2> 5839 T123 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5839 T123 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1417192446160273408)} 0 4
[junit4:junit4]   2> ASYNC  NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@70fd10b5
[junit4:junit4]   2> 5897 T135 C5 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@557ce599 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4fbd7fe9),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5898 T135 C5 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5900 T135 C5 oass.SolrIndexSearcher.<init> Opening Searcher@2a672727 realtime
[junit4:junit4]   2> 5900 T135 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5901 T135 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1417192446170759168)} 0 57
[junit4:junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@77d2eed9
[junit4:junit4]   2> 5908 T149 C6 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@29e012bb lockFactory=org.apache.lucene.store.NativeFSLockFactory@e04184a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5909 T149 C6 oasc.SolrDeletionPolicy.updateCommits newest commit = 1
[junit4:junit4]   2> 5910 T149 C6 oass.SolrIndexSearcher.<init> Opening Searcher@51c9a161 realtime
[junit4:junit4]   2> 5910 T149 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 5910 T149 C6 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1417192446234722304)} 0 4
[junit4:junit4]   2> 5920 T83 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16
[junit4:junit4]   2> 5970 T96 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[1 (1417192446297636864)]} 0 8
[junit4:junit4]   2> 5974 T83 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:30973/solr/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 5995 T95 C2 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6003 T95 C2 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5793a045 lockFactory=org.apache.lucene.store.NativeFSLockFactory@397c8a21),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5793a045 lockFactory=org.apache.lucene.store.NativeFSLockFactory@397c8a21),segFN=segments_2,generation=2,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 6004 T95 C2 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 6017 T95 C2 oass.SolrIndexSearcher.<init> Opening Searcher@54f7541 main
[junit4:junit4]   2> 6017 T95 C2 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6018 T97 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@54f7541 main{StandardDirectoryReader(segments_2:3:nrt _0(4.1):C1)}
[junit4:junit4]   2> 6018 T95 C2 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6018 T95 C2 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 23
[junit4:junit4]   2> 6072 T96 C2 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=1 status=0 QTime=35 
[junit4:junit4]   2> 6079 T108 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&commit_end_point=true&version=2} {add=[2 (1417192446414028800), 3 (1417192446415077376), 4 (1417192446415077377)]} 0 3
[junit4:junit4]   2> 6081 T95 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&commit_end_point=true&version=2} {add=[2 (1417192446415077376), 3 (1417192446418223104)]} 0 3
[junit4:junit4]   2> 6082 T83 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:30973/solr/, StdNode: http://127.0.0.1:30978/solr/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 6085 T96 C2 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6085 T109 C3 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6088 T109 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@309763a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5fed2117),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@309763a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5fed2117),segFN=segments_2,generation=2,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 6088 T109 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 2
[junit4:junit4]   2> 6090 T109 C3 oass.SolrIndexSearcher.<init> Opening Searcher@66457096 main
[junit4:junit4]   2> 6090 T109 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 6090 T111 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@66457096 main{StandardDirectoryReader(segments_2:3:nrt _0(4.1):C3)}
[junit4:junit4]   2> 6090 T109 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6094 T109 C3 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 9
[junit4:junit4]   2> 6097 T96 C2 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5793a045 lockFactory=org.apache.lucene.store.NativeFSLockFactory@397c8a21),segFN=segments_2,generation=2,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5793a045 lockFactory=org.apache.lucene.store.NativeFSLockFactory@397c8a21),segFN=segments_3,generation=3,filenames=[_1_nrm.cfs, _0.fnm, _1.frq, _0_nrm.cfs, _1.fnm, _1_nrm.cfe, _1.fdx, _1.prx, _1.fdt, _1.si, _0_nrm.cfe, _0.tim, _0.frq, _0.fdx, _0.si, _1.tip, _0.tip, _0.prx, _1.tim, segments_3, _0.fdt]
[junit4:junit4]   2> 6098 T96 C2 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 6099 T96 C2 oass.SolrIndexSearcher.<init> Opening Searcher@2c0bf31c main
[junit4:junit4]   2> 6099 T96 C2 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6100 T97 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c0bf31c main{StandardDirectoryReader(segments_3:5:nrt _0(4.1):C1 _1(4.1):C2)}
[junit4:junit4]   2> 6100 T96 C2 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6100 T96 C2 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 15
[junit4:junit4]   2> 6104 T95 C2 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=3 status=0 QTime=1 
[junit4:junit4]   2> 6111 T110 C3 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=3 status=0 QTime=0 
[junit4:junit4]   2> 6119 T96 C2 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&commit_end_point=true&version=2} {delete=[2 (-1417192446457020416)]} 0 2
[junit4:junit4]   2> 6119 T108 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&commit_end_point=true&version=2} {delete=[2 (-1417192446458068992)]} 0 1
[junit4:junit4]   2> 6120 T83 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:30973/solr/, StdNode: http://127.0.0.1:30978/solr/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 6122 T109 C3 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6122 T95 C2 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6135 T109 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@309763a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5fed2117),segFN=segments_2,generation=2,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@309763a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5fed2117),segFN=segments_3,generation=3,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, _0_1.del, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt, segments_3]
[junit4:junit4]   2> 6136 T95 C2 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5793a045 lockFactory=org.apache.lucene.store.NativeFSLockFactory@397c8a21),segFN=segments_3,generation=3,filenames=[_1_nrm.cfs, _0.fnm, _1.frq, _0_nrm.cfs, _1.fnm, _1_nrm.cfe, _1.fdx, _1.prx, _1.fdt, _1.si, _0_nrm.cfe, _0.tim, _0.frq, _0.fdx, _0.si, _1.tip, _0.tip, _0.prx, _1.tim, segments_3, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5793a045 lockFactory=org.apache.lucene.store.NativeFSLockFactory@397c8a21),segFN=segments_4,generation=4,filenames=[_1_nrm.cfs, _0.fnm, _1.frq, _0_nrm.cfs, _1_1.del, _1.fnm, _1_nrm.cfe, _1.fdx, _1.prx, _1.fdt, _1.si, _0_nrm.cfe, _0.tim, _0.frq, _0.fdx, _0.si, _1.tip, _0.tip, _0.prx, _1.tim, _0.fdt, segments_4]
[junit4:junit4]   2> 6136 T109 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 3
[junit4:junit4]   2> 6136 T95 C2 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 6136 T109 C3 oass.SolrIndexSearcher.<init> Opening Searcher@5376f412 main
[junit4:junit4]   2> 6136 T95 C2 oass.SolrIndexSearcher.<init> Opening Searcher@23bff716 main
[junit4:junit4]   2> 6137 T109 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 6137 T95 C2 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6137 T111 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5376f412 main{StandardDirectoryReader(segments_3:4:nrt _0(4.1):C3/1)}
[junit4:junit4]   2> 6138 T95 C2 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6138 T97 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23bff716 main{StandardDirectoryReader(segments_4:6:nrt _0(4.1):C1 _1(4.1):C2/1)}
[junit4:junit4]   2> 6137 T109 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6138 T95 C2 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 16
[junit4:junit4]   2> 6138 T109 C3 UPDATE [collection1] webapp=/solr path=/update params={waitSearcher=true&commit=true&commit_end_point=true&expungeDeletes=false&wt=javabin&softCommit=false&version=2} {commit=} 0 16
[junit4:junit4]   2> 6142 T96 C2 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=2 status=0 QTime=1 
[junit4:junit4]   2> 6145 T110 C3 REQ [collection1] webapp=/solr path=/select params={wt=javabin&q=*:*&version=2} hits=2 status=0 QTime=0 
[junit4:junit4]   2> 6149 T108 C3 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6154 T108 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@309763a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5fed2117),segFN=segments_3,generation=3,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, _0_1.del, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt, segments_3]
[junit4:junit4]   2> 		commit{dir=BaseDirectoryWrapper(org.apache.lucene.store.RAMDirectory@309763a8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5fed2117),segFN=segments_4,generation=4,filenames=[_1.fdx, _1.prx, _1_nrm.cfs, _1.frq, _1.fdt, _1.si, _1.tip, _1.fnm, _1.tim, _1_nrm.cfe, segments_4]
[junit4:junit4]   2> 6154 T108 C3 oasc.SolrDeletionPolicy.updateCommits newest commit = 4
[junit4:junit4]   2> 6156 T108 C3 oass.SolrIndexSearcher.<init> Opening Searcher@6b3da2f0 main
[junit4:junit4]   2> 6156 T108 C3 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.BaseDirectoryWrapper
[junit4:junit4]   2> 6156 T108 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6156 T111 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b3da2f0 main{StandardDirectoryReader(segments_4:6:nrt _1(4.1):C2)}
[junit4:junit4]   2> 6157 T108 C3 UPDATE [collection1] webapp=/solr path=/update params={optimize=true&maxSegments=1&waitSearcher=true&wt=javabin&version=2} {optimize=} 0 8
[junit4:junit4]   2> 6160 T121 C4 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6162 T121 C4 oass.SolrIndexSearcher.<init> Opening Searcher@1e2639 main
[junit4:junit4]   2> 6162 T121 C4 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6162 T125 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e2639 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 6162 T121 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6163 T121 C4 UPDATE [collection1] webapp=/solr path=/update params={optimize=true&maxSegments=1&waitSearcher=true&wt=javabin&version=2} {optimize=} 0 3
[junit4:junit4]   2> 6166 T137 C5 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6167 T137 C5 oass.SolrIndexSearcher.<init> Opening Searcher@50a3b5e5 main
[junit4:junit4]   2> 6167 T137 C5 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6168 T139 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@50a3b5e5 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 6168 T137 C5 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6168 T137 C5 UPDATE [collection1] webapp=/solr path=/update params={optimize=true&maxSegments=1&waitSearcher=true&wt=javabin&version=2} {optimize=} 0 3
[junit4:junit4]   2> 6171 T152 C6 oasu.DirectUpdateHandler2.commit start commit{flags=0,_version_=0,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false}
[junit4:junit4]   2> 6173 T152 C6 oass.SolrIndexSearcher.<init> Opening Searcher@73cbbf6d main
[junit4:junit4]   2> 6173 T152 C6 oass.SolrIndexSearcher.getIndexDir WARNING WARNING: Directory impl does not support setting indexDir: org.apache.lucene.store.MockDirectoryWrapper
[junit4:junit4]   2> 6173 T153 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73cbbf6d main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 6173 T152 C6 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 6174 T152 C6 UPDATE [collection1] webapp=/solr path=/update params={optimize=true&maxSegments=1&waitSearcher=true&wt=javabin&version=2} {optimize=} 0 3
[junit4:junit4]   2> 6190 T108 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[25 (1417192446522032128), 26 (1417192446525177856), 27 (1417192446525177857), 28 (1417192446526226432), 29 (1417192446527275008), 30 (1417192446527275009), 34 (1417192446528323584), 35 (1417192446528323585), 36 (1417192446529372160), 39 (1417192446530420736)]} 0 10
[junit4:junit4]   2> 6212 T109 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5 (1417192446519934976), 6 (1417192446532517888), 7 (1417192446537760768), 9 (1417192446544052224), 10 (1417192446546149376), 12 (1417192446547197953), 13 (1417192446549295106), 16 (1417192446551392256), 18 (1417192446552440834), 23 (1417192446553489409)]} 0 34
[junit4:junit4]   2> 6219 T138 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5 (1417192446520983552), 7 (1417192446537760768), 9 (1417192446538809344), 13 (1417192446545100800), 14 (1417192446548246528), 15 (1417192446550343680), 16 (1417192446552440833), 19 (1417192446555586561), 23 (1417192446556635136), 25 (1417192446561878016)]} 0 41
[junit4:junit4]   2> 6222 T124 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[7 (1417192446519934976), 8 (1417192446532517888), 12 (1417192446538809345), 14 (1417192446544052226), 16 (1417192446547197952), 19 (1417192446548246528), 20 (1417192446549295106), 21 (1417192446551392256), 24 (1417192446555586561), 25 (1417192446557683712)]} 0 44
[junit4:junit4]   2> 6222 T121 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[27 (1417192446523080704), 29 (1417192446538809344), 30 (1417192446544052224), 31 (1417192446546149376), 37 (1417192446548246529), 38 (1417192446550343680), 40 (1417192446552440833), 41 (1417192446555586560), 44 (1417192446556635139), 46 (1417192446561878018)]} 0 41
[junit4:junit4]   2> 6229 T151 C6 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[26 (1417192446523080704), 27 (1417192446538809344), 29 (1417192446543003649), 30 (1417192446548246528), 31 (1417192446549295105), 32 (1417192446551392257), 33 (1417192446557683712), 36 (1417192446565023745), 38 (1417192446569218048), 40 (1417192446570266626)]} 0 49
[junit4:junit4]   2> 6229 T110 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[45 (1417192446543003648), 46 (1417192446545100800), 48 (1417192446547197952), 49 (1417192446549295104), 51 (1417192446550343680), 57 (1417192446552440835), 58 (1417192446554537984), 59 (1417192446565023744), 61 (1417192446566072320), 63 (1417192446569218048)]} 0 33
[junit4:junit4]   2> 6230 T123 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[47 (1417192446544052225), 49 (1417192446547197953), 50 (1417192446549295104), 53 (1417192446551392257), 54 (1417192446553489408), 55 (1417192446556635138), 56 (1417192446561878016), 59 (1417192446562926593), 61 (1417192446566072320), 62 (1417192446573412352)]} 0 29
[junit4:junit4]   2> 6230 T108 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[64 (1417192446549295105), 67 (1417192446552440832), 68 (1417192446553489408), 69 (1417192446555586561), 71 (1417192446557683712), 72 (1417192446565023745), 74 (1417192446568169473), 75 (1417192446569218050), 76 (1417192446570266626), 77 (1417192446571315200)]} 0 24
[junit4:junit4]   2> 6231 T149 C6 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[5 (1417192446519934976), 6 (1417192446537760768), 7 (1417192446545100800), 12 (1417192446547197952), 13 (1417192446549295104), 14 (1417192446551392256), 19 (1417192446562926592), 21 (1417192446563975168), 22 (1417192446565023744), 24 (1417192446570266624)]} 0 54
[junit4:junit4]   2> 6231 T172 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[66 (1417192446549295105), 72 (1417192446552440832), 73 (1417192446553489409), 74 (1417192446556635137), 76 (1417192446558732288), 80 (1417192446562926592), 81 (1417192446566072321), 84 (1417192446569218048), 85 (1417192446569218051), 86 (1417192446570266624)]} 0 25
[junit4:junit4]   2> 6233 T175 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[78 (1417192446552440833), 80 (1417192446555586560), 82 (1417192446556635136), 83 (1417192446565023746), 84 (1417192446567120896), 86 (1417192446569218049), 87 (1417192446574460928), 88 (1417192446575509504), 89 (1417192446575509506), 92 (1417192446576558081)]} 0 24
[junit4:junit4]   2> 6233 T170 C6 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[64 (1417192446549295106), 65 (1417192446555586560), 66 (1417192446563975171), 68 (1417192446566072320), 70 (1417192446568169472), 73 (1417192446570266628), 74 (1417192446574460928), 78 (1417192446575509506), 80 (1417192446576558081), 82 (1417192446576558083)]} 0 27
[junit4:junit4]   2> 6235 T188 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[111 (1417192446559780864), 112 (1417192446568169473), 114 (1417192446569218050), 116 (1417192446575509504), 117 (1417192446575509505), 118 (1417192446576558080), 120 (1417192446577606656), 122 (1417192446577606657), 123 (1417192446578655232), 124 (1417192446578655233)]} 0 20
[junit4:junit4]   2> 6238 T137 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[26 (1417192446544052224), 31 (1417192446546149377), 32 (1417192446549295104), 36 (1417192446552440832), 38 (1417192446553489408), 39 (1417192446555586564), 40 (1417192446558732289), 41 (1417192446566072321), 42 (1417192446569218050), 47 (1417192446570266625)]} 0 38
[junit4:junit4]   2> 6241 T183 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[95 (1417192446556635137), 97 (1417192446561878016), 98 (1417192446562926592), 102 (1417192446563975168), 103 (1417192446568169472), 104 (1417192446570266625), 105 (1417192446573412352), 106 (1417192446574460929), 110 (1417192446582849537), 114 (1417192446584946690)]} 0 28
[junit4:junit4]   2> 6242 T176 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[77 (1417192446551392256), 78 (1417192446555586560), 79 (1417192446558732288), 80 (1417192446562926592), 83 (1417192446565023745), 86 (1417192446569218048), 88 (1417192446573412353), 89 (1417192446583898112), 90 (1417192446584946692), 95 (1417192446585995266)]} 0 34
[junit4:junit4]   2> 6243 T182 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[89 (1417192446556635136), 92 (1417192446561878017), 94 (1417192446563975168), 97 (1417192446568169472), 98 (1417192446569218049), 99 (1417192446573412354), 103 (1417192446582849536), 105 (1417192446583898113), 109 (1417192446584946689), 110 (1417192446585995265)]} 0 31
[junit4:junit4]   2> 6244 T110 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[140 (1417192446575509505), 142 (1417192446575509507), 143 (1417192446576558080), 144 (1417192446577606656), 146 (1417192446578655232), 147 (1417192446578655233), 150 (1417192446579703808), 151 (1417192446583898113), 152 (1417192446585995265), 153 (1417192446588092417)]} 0 13
[junit4:junit4]   2> 6250 T189 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[114 (1417192446566072320), 117 (1417192446570266624), 119 (1417192446573412352), 120 (1417192446573412355), 122 (1417192446583898113), 123 (1417192446584946691), 124 (1417192446587043840), 125 (1417192446588092417), 126 (1417192446589140992), 127 (1417192446594383872)]} 0 27
[junit4:junit4]   2> 6251 T108 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[154 (1417192446578655234), 157 (1417192446579703809), 162 (1417192446583898112), 167 (1417192446584946689), 170 (1417192446585995264), 173 (1417192446587043840), 174 (1417192446588092416), 175 (1417192446589140992), 176 (1417192446594383873), 180 (1417192446595432450)]} 0 17
[junit4:junit4]   2> 6252 T135 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[48 (1417192446544052225), 49 (1417192446546149376), 55 (1417192446547197952), 59 (1417192446550343681), 60 (1417192446554537984), 64 (1417192446555586563), 65 (1417192446568169472), 68 (1417192446574460928), 70 (1417192446593335296), 74 (1417192446596481024)]} 0 51
[junit4:junit4]   2> 6252 T109 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[117 (1417192446570266624), 119 (1417192446570266627), 121 (1417192446573412353), 125 (1417192446582849536), 131 (1417192446584946688), 132 (1417192446585995267), 133 (1417192446587043842), 135 (1417192446594383872), 137 (1417192446595432449), 139 (1417192446596481024)]} 0 27
[junit4:junit4]   2> 6256 T181 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[97 (1417192446555586562), 98 (1417192446565023744), 99 (1417192446569218049), 101 (1417192446575509504), 102 (1417192446584946689), 103 (1417192446585995264), 106 (1417192446587043841), 107 (1417192446588092418), 111 (1417192446589140994), 112 (1417192446595432448)]} 0 43
[junit4:junit4]   2> 6258 T152 C6 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[41 (1417192446543003648), 43 (1417192446546149376), 45 (1417192446550343680), 46 (1417192446563975169), 49 (1417192446570266629), 51 (1417192446575509505), 54 (1417192446575509507), 56 (1417192446576558082), 59 (1417192446584946691), 61 (1417192446585995265)]} 0 62
[junit4:junit4]   2> 6260 T191 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[125 (1417192446573412353), 126 (1417192446583898112), 127 (1417192446584946688), 130 (1417192446585995264), 131 (1417192446587043840), 132 (1417192446588092416), 133 (1417192446589140992), 134 (1417192446601723905), 137 (1417192446603821059), 140 (1417192446604869635)]} 0 31
[junit4:junit4]   2> 6267 T138 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[128 (1417192446573412354), 129 (1417192446582849536), 130 (1417192446583898114), 132 (1417192446584946690), 133 (1417192446585995265), 134 (1417192446588092416), 135 (1417192446589140993), 137 (1417192446602772480), 139 (1417192446606966787), 140 (1417192446612209664)]} 0 38
[junit4:junit4]   2> 6269 T175 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[203 (1417192446585995266), 205 (1417192446587043841), 206 (1417192446588092418), 209 (1417192446589140993), 213 (1417192446596481025), 215 (1417192446597529600), 219 (1417192446601723904), 220 (1417192446605918208), 221 (1417192446606966784), 223 (1417192446613258240)]} 0 28
[junit4:junit4]   2> 6269 T188 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[144 (1417192446594383874), 145 (1417192446595432450), 146 (1417192446596481026), 149 (1417192446597529603), 150 (1417192446601723907), 151 (1417192446606966784), 156 (1417192446611161088), 159 (1417192446612209666), 162 (1417192446613258241), 165 (1417192446613258246)]} 0 20
[junit4:junit4]   2> 6271 T179 C6 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[85 (1417192446553489408), 87 (1417192446563975170), 88 (1417192446569218049), 90 (1417192446570266627), 91 (1417192446575509504), 93 (1417192446584946688), 96 (1417192446585995264), 97 (1417192446587043841), 98 (1417192446593335296), 101 (1417192446612209665)]} 0 61
[junit4:junit4]   2> 6272 T124 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[218 (1417192446594383873), 219 (1417192446595432448), 221 (1417192446595432449), 224 (1417192446596481025), 225 (1417192446597529601), 227 (1417192446597529604), 228 (1417192446602772480), 231 (1417192446603821058), 232 (1417192446611161089), 234 (1417192446617452544)]} 0 23
[junit4:junit4]   2> 6272 T149 C6 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[181 (1417192446582849536), 182 (1417192446583898112), 183 (1417192446584946689), 187 (1417192446587043840), 189 (1417192446587043842), 190 (1417192446589140992), 191 (1417192446594383872), 192 (1417192446595432448), 196 (1417192446613258240), 197 (1417192446617452544)]} 0 37
[junit4:junit4]   2> 6272 T123 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[235 (1417192446597529602), 237 (1417192446603821056), 239 (1417192446603821057), 241 (1417192446604869633), 242 (1417192446605918208), 243 (1417192446606966785), 246 (1417192446613258240), 247 (1417192446615355392), 249 (1417192446616403970), 254 (1417192446617452548)]} 0 20
[junit4:junit4]   2> 6274 T182 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[205 (1417192446596481024), 206 (1417192446597529600), 207 (1417192446601723904), 208 (1417192446611161090), 210 (1417192446612209668), 211 (1417192446613258244), 212 (1417192446615355394), 213 (1417192446617452546), 215 (1417192446618501120), 216 (1417192446619549697)]} 0 24
[junit4:junit4]   2> 6278 T172 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[181 (1417192446594383872), 186 (1417192446601723906), 187 (1417192446612209664), 191 (1417192446613258245), 193 (1417192446616403968), 196 (1417192446617452545), 197 (1417192446617452547), 201 (1417192446618501122), 202 (1417192446619549698), 203 (1417192446620598272)]} 0 30
[junit4:junit4]   2> 6279 T137 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[213 (1417192446594383873), 217 (1417192446595432449), 218 (1417192446596481025), 220 (1417192446597529600), 222 (1417192446600675328), 223 (1417192446605918208), 224 (1417192446611161088), 225 (1417192446613258244), 229 (1417192446618501121), 231 (1417192446624792578)]} 0 30
[junit4:junit4]   2> 6298 T207 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[190 (1417192446601723904), 191 (1417192446613258242), 192 (1417192446615355392), 196 (1417192446620598272), 198 (1417192446623744002), 201 (1417192446624792579), 205 (1417192446625841153), 207 (1417192446639472640), 209 (1417192446640521219), 211 (1417192446644715520)]} 0 43
[junit4:junit4]   2> 6298 T181 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[249 (1417192446604869632), 251 (1417192446606966784), 254 (1417192446613258243), 257 (1417192446619549698), 259 (1417192446623744000), 260 (1417192446624792576), 261 (1417192446625841152), 263 (1417192446626889729), 265 (1417192446640521216), 267 (1417192446645764096)]} 0 39
[junit4:junit4]   2> 6299 T151 C6 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[161 (1417192446594383874), 162 (1417192446595432449), 163 (1417192446596481025), 165 (1417192446597529600), 169 (1417192446603821056), 172 (1417192446604869633), 176 (1417192446604869635), 177 (1417192446613258241), 178 (1417192446616403968), 180 (1417192446645764096)]} 0 50
[junit4:junit4]   2> 6300 T121 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[167 (1417192446588092417), 168 (1417192446589140993), 169 (1417192446602772481), 171 (1417192446604869634), 172 (1417192446605918209), 174 (1417192446612209665), 175 (1417192446613258243), 176 (1417192446616403969), 177 (1417192446624792577), 179 (1417192446640521218)]} 0 57
[junit4:junit4]   2> 6301 T205 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[224 (1417192446600675329), 225 (1417192446605918209), 226 (1417192446611161089), 227 (1417192446617452544), 231 (1417192446618501120), 234 (1417192446619549696), 236 (1417192446620598272), 237 (1417192446640521216), 240 (1417192446641569792), 241 (1417192446647861248)]} 0 46
[junit4:junit4]   2> 6302 T206 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[184 (1417192446595432448), 189 (1417192446600675328), 190 (1417192446605918210), 191 (1417192446611161088), 194 (1417192446614306816), 196 (1417192446616403969), 198 (1417192446617452546), 199 (1417192446623744000), 200 (1417192446646812672), 201 (1417192446647861249)]} 0 52
[junit4:junit4]   2> 6302 T215 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[255 (1417192446604869632), 257 (1417192446612209667), 260 (1417192446613258242), 261 (1417192446615355393), 262 (1417192446624792578), 264 (1417192446625841152), 266 (1417192446639472640), 267 (1417192446640521216), 268 (1417192446647861249), 269 (1417192446648909826)]} 0 43
[junit4:junit4]   2> 6304 T188 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[270 (1417192446617452549), 271 (1417192446618501121), 275 (1417192446619549696), 281 (1417192446623744000), 282 (1417192446639472641), 283 (1417192446640521219), 286 (1417192446647861251), 287 (1417192446648909828), 288 (1417192446649958402), 291 (1417192446651006979)]} 0 33
[junit4:junit4]   2> 6305 T108 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[283 (1417192446640521217), 286 (1417192446645764096), 288 (1417192446645764100), 289 (1417192446648909825), 290 (1417192446648909826), 291 (1417192446649958400), 292 (1417192446651006976), 294 (1417192446651006978), 295 (1417192446652055553), 297 (1417192446652055555)]} 0 13
[junit4:junit4]   2> 6309 T198 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[141 (1417192446577606656), 144 (1417192446605918209), 149 (1417192446613258240), 151 (1417192446618501120), 152 (1417192446619549697), 153 (1417192446620598273), 157 (1417192446640521218), 158 (1417192446651006976), 160 (1417192446652055556), 161 (1417192446656249857)]} 0 77
[junit4:junit4]   2> 6309 T183 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[244 (1417192446604869632), 245 (1417192446606966785), 252 (1417192446616403968), 254 (1417192446617452545), 257 (1417192446618501121), 258 (1417192446619549697), 260 (1417192446623744002), 263 (1417192446640521219), 264 (1417192446645764097), 265 (1417192446645764099)]} 0 50
[junit4:junit4]   2> 6310 T176 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[235 (1417192446600675329), 236 (1417192446606966786), 237 (1417192446614306816), 238 (1417192446626889728), 239 (1417192446640521217), 241 (1417192446652055554), 243 (1417192446653104130), 244 (1417192446654152706), 245 (1417192446656249856), 247 (1417192446657298435)]} 0 55
[junit4:junit4]   2> 6310 T170 C6 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[143 (1417192446594383873), 145 (1417192446596481024), 146 (1417192446604869632), 149 (1417192446605918208), 150 (1417192446606966784), 152 (1417192446611161089), 154 (1417192446624792577), 156 (1417192446625841152), 157 (1417192446640521217), 159 (1417192446653104128)]} 0 61
[junit4:junit4]   2> 6310 T186 C6 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[103 (1417192446567120896), 106 (1417192446570266625), 107 (1417192446576558080), 108 (1417192446584946690), 109 (1417192446596481026), 110 (1417192446597529601), 111 (1417192446601723905), 118 (1417192446611161088), 121 (1417192446620598272), 122 (1417192446641569792)]} 0 90
[junit4:junit4]   2> 6465 T205 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[368 (1417192446656249856), 370 (1417192446810390530), 371 (1417192446812487682), 372 (1417192446813536258), 373 (1417192446814584835), 376 (1417192446815633411), 377 (1417192446816681987), 379 (1417192446817730560), 381 (1417192446818779136), 382 (1417192446819827713)]} 0 157
[junit4:junit4]   2> 6461 T175 C3 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[348 (1417192446648909824), 354 (1417192446648909827), 355 (1417192446810390528), 356 (1417192446811439105), 358 (1417192446812487683), 359 (1417192446813536256), 360 (1417192446813536260), 361 (1417192446814584834), 362 (1417192446815633410), 367 (1417192446816681985)]} 0 160
[junit4:junit4]   2> 6466 T176 C5 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[367 (1417192446808293377), 369 (1417192446809341952), 370 (1417192446810390530), 371 (1417192446812487681), 374 (1417192446813536256), 376 (1417192446814584833), 377 (1417192446815633408), 379 (1417192446818779138), 381 (1417192446819827712), 382 (1417192446820876288)]} 0 13
[junit4:junit4]   2> 6460 T191 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[292 (1417192446624792576), 294 (1417192446640521217), 295 (1417192446647861250), 298 (1417192446649958400), 300 (1417192446651006978), 303 (1417192446652055553), 305 (1417192446656249856), 306 (1417192446659395584), 312 (1417192446812487680), 313 (1417192446814584833)]} 0 183
[junit4:junit4]   2> 6460 T123 C4 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[351 (1417192446648909824), 354 (1417192446648909829), 357 (1417192446651006977), 358 (1417192446651006981), 361 (1417192446653104128), 362 (141719244680829

[...truncated too long message...]

 org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   11) Thread[id=120, name=qtp829021858-120 Acceptor3 SelectChannelConnector@0.0.0.0:30979, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   12) Thread[id=131, name=qtp779119119-131 Acceptor0 SelectChannelConnector@0.0.0.0:30980, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   13) Thread[id=119, name=qtp829021858-119 Acceptor2 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   14) Thread[id=92, name=qtp813186733-92 Acceptor2 SelectChannelConnector@0.0.0.0:30973, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   15) Thread[id=147, name=qtp170949019-147 Acceptor2 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   16) Thread[id=104, name=qtp680387598-104 Acceptor1 SelectChannelConnector@0.0.0.0:30978, state=RUNNABLE, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   17) Thread[id=118, name=qtp829021858-118 Acceptor1 SelectChannelConnector@0.0.0.0:30979, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   18) Thread[id=105, name=qtp680387598-105 Acceptor2 SelectChannelConnector@0.0.0.0:30978, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   19) Thread[id=146, name=qtp170949019-146 Acceptor1 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    >   20) Thread[id=148, name=qtp170949019-148 Acceptor3 SelectChannelConnector@0.0.0.0:30981, state=BLOCKED, group=TGRP-SolrCmdDistributorTest]
[junit4:junit4]    >         at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:147)
[junit4:junit4]    >         at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:104)
[junit4:junit4]    >         at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:933)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
[junit4:junit4]    >         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([C51BE694BBAD5B5B]:0)
[junit4:junit4] Completed on J0 in 31.92s, 1 test, 2 errors <<< FAILURES!

[...truncated 173 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:335: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:178: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/module-build.xml:61: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1159: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:827: There were test failures: 233 suites, 965 tests, 4 suite-level errors, 855 ignored

Total time: 19 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure