You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2016/07/19 12:03:33 UTC

ZooKeeper-trunk-jdk8 - Build # 673 - Still Failing

See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/673/

###################################################################################
########################## LAST 60 LINES OF THE CONSOLE ###########################
[...truncated 404019 lines...]
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
    [junit] 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
    [junit] 2016-07-19 11:59:09,284 [myid:] - INFO  [ProcessThread(sid:0 cport:19545)::PrepRequestProcessor@647] - Processed session termination for sessionid: 0x1015c3b7a760000
    [junit] 2016-07-19 11:59:09,298 [myid:] - WARN  [New I/O worker #6635:NettyServerCnxnFactory$CnxnChannelHandler@142] - Exception caught [id: 0xe6591d41, /127.0.0.1:43051 :> /127.0.0.1:19545] EXCEPTION: java.nio.channels.ClosedChannelException
    [junit] java.nio.channels.ClosedChannelException
    [junit] 	at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:265)
    [junit] 	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:456)
    [junit] 	at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$UnpooledSendBuffer.transferTo(SocketSendBufferPool.java:203)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.write0(AbstractNioWorker.java:201)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromTaskLoop(AbstractNioWorker.java:151)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioChannel$WriteTask.run(AbstractNioChannel.java:315)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    [junit] 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    [junit] 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    [junit] 	at java.lang.Thread.run(Thread.java:744)
    [junit] 2016-07-19 11:59:09,298 [myid:] - INFO  [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port19545,name1=Connections,name2=127.0.0.1,name3=0x1015c3b7a760000]
    [junit] 2016-07-19 11:59:09,399 [myid:] - INFO  [main:ZooKeeper@1313] - Session: 0x1015c3b7a760000 closed
    [junit] 2016-07-19 11:59:09,399 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 150690
    [junit] 2016-07-19 11:59:09,399 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 1641
    [junit] 2016-07-19 11:59:09,399 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testWatcherAutoResetWithLocal
    [junit] 2016-07-19 11:59:09,399 [myid:] - INFO  [main:ClientBase@543] - tearDown starting
    [junit] 2016-07-19 11:59:09,399 [myid:] - INFO  [main:ClientBase@513] - STOPPING server
    [junit] 2016-07-19 11:59:09,399 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x1015c3b7a760000
    [junit] 2016-07-19 11:59:09,400 [myid:] - INFO  [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:19545
    [junit] 2016-07-19 11:59:09,403 [myid:] - INFO  [main:ZooKeeperServer@498] - shutting down
    [junit] 2016-07-19 11:59:09,403 [myid:] - INFO  [main:SessionTrackerImpl@232] - Shutting down
    [junit] 2016-07-19 11:59:09,404 [myid:] - INFO  [main:PrepRequestProcessor@965] - Shutting down
    [junit] 2016-07-19 11:59:09,404 [myid:] - INFO  [main:SyncRequestProcessor@191] - Shutting down
    [junit] 2016-07-19 11:59:09,404 [myid:] - INFO  [ProcessThread(sid:0 cport:19545)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
    [junit] 2016-07-19 11:59:09,404 [myid:] - INFO  [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
    [junit] 2016-07-19 11:59:09,405 [myid:] - INFO  [main:FinalRequestProcessor@479] - shutdown of request processor complete
    [junit] 2016-07-19 11:59:09,405 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port19545,name1=InMemoryDataTree]
    [junit] 2016-07-19 11:59:09,405 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port19545]
    [junit] 2016-07-19 11:59:09,405 [myid:] - INFO  [main:FourLetterWordMain@85] - connecting to 127.0.0.1 19545
    [junit] 2016-07-19 11:59:09,406 [myid:] - INFO  [main:JMXEnv@146] - ensureOnly:[]
    [junit] 2016-07-19 11:59:09,410 [myid:] - INFO  [main:ClientBase@568] - fdcount after test is: 4827 at start it was 4831
    [junit] 2016-07-19 11:59:09,411 [myid:] - INFO  [main:ZKTestCase$1@65] - SUCCEEDED testWatcherAutoResetWithLocal
    [junit] 2016-07-19 11:59:09,411 [myid:] - INFO  [main:ZKTestCase$1@60] - FINISHED testWatcherAutoResetWithLocal
    [junit] Tests run: 101, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 489.391 sec, Thread: 4, Class: org.apache.zookeeper.test.NioNettySuiteTest
    [junit] Running org.apache.zookeeper.server.quorum.Zab1_0Test in thread 6
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec, Thread: 6, Class: org.apache.zookeeper.server.quorum.Zab1_0Test
    [junit] Test org.apache.zookeeper.server.quorum.Zab1_0Test FAILED (timeout)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build.xml:1207: Tests failed!

Total time: 16 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###################################################################################
############################## FAILED TESTS (if any) ##############################
29 tests failed.
FAILED:  org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalObserverRun

Error Message:
Timeout occurred. Please note the time in the report does not reflect the time until the timeout.

Stack Trace:
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.MultiTransactionTest.testUpdateConflict[1]

Error Message:
Unable to connect to server

Stack Trace:
junit.framework.AssertionFailedError: Unable to connect to server
	at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:188)
	at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:177)
	at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:162)
	at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:155)
	at org.apache.zookeeper.test.MultiTransactionTest.setUp(MultiTransactionTest.java:82)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testHammer

Error Message:
null

Stack Trace:
junit.framework.AssertionFailedError
	at org.apache.zookeeper.test.AsyncHammerTest.testHammer(AsyncHammerTest.java:186)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer(AsyncHammerTest.java:208)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncCreate2Failure_NoNode

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncCreateFailure_NoNode

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.<init>(ClientCommunicatorAdmin.java:45)
	at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.<init>(RMIConnector.java:1467)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:339)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncSetACL

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncMultiFailure_NoSideEffect

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.<init>(ClientCommunicatorAdmin.java:45)
	at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.<init>(RMIConnector.java:1467)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:339)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncGetData

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncSync

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncGetChildren2Two

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncCreateFailure_NoChildForEphemeral

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncDeleteFailure_BadVersion

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncDeleteFailure_NoNode

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncSetDataFailure_BadVersion

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncGetChildrenFailure_NoNode

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncCreate2

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncExistsFailure_NoNode

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncGetChildren2Empty

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncCreateThree

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncMultiFailure_AllErrorResult

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncGetChildrenTwo

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncCreateFailure_NodeExists

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncCreate2Failure_NoChildForEphemeral

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncSetData

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncSetDataFailure_NoNode

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncGetACL

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.NioServerBoss.<init>(NioServerBoss.java:49)
	at org.jboss.netty.channel.socket.nio.NioServerBossPool.newBoss(NioServerBossPool.java:55)
	at org.jboss.netty.channel.socket.nio.NioServerBossPool.newBoss(NioServerBossPool.java:26)
	at org.jboss.netty.channel.socket.nio.AbstractNioBossPool.init(AbstractNioBossPool.java:75)
	at org.jboss.netty.channel.socket.nio.NioServerBossPool.<init>(NioServerBossPool.java:40)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:179)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.quorum.QuorumPeer.<init>(QuorumPeer.java:863)
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:175)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.AsyncHammerTest.setUp(AsyncHammerTest.java:52)
	at org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer(AsyncHammerTest.java:203)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testHammer

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.quorum.QuorumPeer.<init>(QuorumPeer.java:863)
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:184)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.AsyncHammerTest.setUp(AsyncHammerTest.java:52)
	at org.apache.zookeeper.test.AsyncHammerTest.testHammer(AsyncHammerTest.java:171)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)



ZooKeeper-trunk-jdk8 - Build # 680 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/680/

###################################################################################
########################## LAST 60 LINES OF THE CONSOLE ###########################
[...truncated 459345 lines...]
    [junit] java.net.ConnectException: Connection refused: 127.0.0.1/127.0.0.1:14036
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    [junit] 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    [junit] 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    [junit] 	at java.lang.Thread.run(Thread.java:744)
    [junit] 2016-07-26 11:57:26,743 [myid:] - INFO  [New I/O boss #9405:ClientCnxnSocketNetty@208] - channel is told closing
    [junit] 2016-07-26 11:57:26,745 [myid:127.0.0.1:14036] - INFO  [main-SendThread(127.0.0.1:14036):ClientCnxn$SendThread@1231] - channel for sessionid 0x1012c5a340a0000 is lost, closing socket connection and attempting reconnect
    [junit] 2016-07-26 11:57:27,296 [myid:127.0.0.1:14042] - INFO  [main-SendThread(127.0.0.1:14042):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:14042. Will not attempt to authenticate using SASL (unknown error)
    [junit] 2016-07-26 11:57:27,297 [myid:] - INFO  [New I/O boss #9471:ClientCnxnSocketNetty$1@127] - future isn't success, cause: {}
    [junit] java.net.ConnectException: Connection refused: 127.0.0.1/127.0.0.1:14042
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    [junit] 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    [junit] 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    [junit] 	at java.lang.Thread.run(Thread.java:744)
    [junit] 2016-07-26 11:57:27,297 [myid:] - WARN  [New I/O boss #9471:ClientCnxnSocketNetty$ZKClientHandler@439] - Exception caught: [id: 0x3ae7f050] EXCEPTION: java.net.ConnectException: Connection refused: 127.0.0.1/127.0.0.1:14042
    [junit] java.net.ConnectException: Connection refused: 127.0.0.1/127.0.0.1:14042
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    [junit] 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    [junit] 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    [junit] 	at java.lang.Thread.run(Thread.java:744)
    [junit] 2016-07-26 11:57:27,297 [myid:] - INFO  [New I/O boss #9471:ClientCnxnSocketNetty@208] - channel is told closing
    [junit] 2016-07-26 11:57:27,297 [myid:127.0.0.1:14042] - INFO  [main-SendThread(127.0.0.1:14042):ClientCnxn$SendThread@1231] - channel for sessionid 0x3012c5a34730000 is lost, closing socket connection and attempting reconnect
    [junit] Running org.apache.zookeeper.server.quorum.Zab1_0Test in thread 8
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec, Thread: 8, Class: org.apache.zookeeper.server.quorum.Zab1_0Test
    [junit] Test org.apache.zookeeper.server.quorum.Zab1_0Test FAILED (timeout)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build.xml:1207: Tests failed!

Total time: 16 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###################################################################################
############################## FAILED TESTS (if any) ##############################
1 tests failed.
FAILED:  org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalObserverRun

Error Message:
Timeout occurred. Please note the time in the report does not reflect the time until the timeout.

Stack Trace:
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
	at java.lang.Thread.run(Thread.java:744)



ZooKeeper-trunk-jdk8 - Build # 679 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/679/

###################################################################################
########################## LAST 60 LINES OF THE CONSOLE ###########################
[...truncated 383214 lines...]
    [junit] 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    [junit] 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    [junit] 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    [junit] 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
    [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
    [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
    [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
    [junit] 2016-07-25 12:00:21,458 [myid:] - INFO  [main:ZKTestCase$1@60] - FINISHED testAsyncGetACL
    [junit] 2016-07-25 12:00:21,460 [myid:] - INFO  [main:PortAssignment@85] - Assigned port 27391 from range 27379 - 30071.
    [junit] 2016-07-25 12:00:21,460 [myid:] - INFO  [main:ZKTestCase$1@55] - STARTING testAsyncCreate2Failure_NoNode
    [junit] 2016-07-25 12:00:21,461 [myid:] - INFO  [main:ClientBase@448] - Initial fdcount is: 59
    [junit] 2016-07-25 12:02:21,581 [myid:] - INFO  [main:ZKTestCase$1@70] - FAILED testAsyncCreate2Failure_NoNode
    [junit] org.junit.internal.runners.model.MultipleFailureException: There were 2 errors:
    [junit]   java.rmi.ConnectIOException(error during JRMP connection establishment; nested exception is: 
    [junit] 	java.net.SocketTimeoutException: Read timed out)
    [junit]   java.lang.NullPointerException(null)
    [junit] 	at org.junit.runners.model.MultipleFailureException.assertEmpty(MultipleFailureException.java:67)
    [junit] 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:39)
    [junit] 	at org.junit.rules.TestWatchman$1.evaluate(TestWatchman.java:53)
    [junit] 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    [junit] 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    [junit] 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
    [junit] 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    [junit] 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    [junit] 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    [junit] 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    [junit] 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    [junit] 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    [junit] 	at org.junit.runners.Suite.runChild(Suite.java:128)
    [junit] 	at org.junit.runners.Suite.runChild(Suite.java:27)
    [junit] 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    [junit] 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    [junit] 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    [junit] 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    [junit] 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    [junit] 	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    [junit] 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    [junit] 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    [junit] 	at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:38)
    [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:532)
    [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1179)
    [junit] 	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:1030)
    [junit] 2016-07-25 12:02:21,582 [myid:] - INFO  [main:ZKTestCase$1@60] - FINISHED testAsyncCreate2Failure_NoNode
    [junit] 2016-07-25 12:02:21,583 [myid:] - INFO  [main:PortAssignment@85] - Assigned port 27392 from range 27379 - 30071.
    [junit] 2016-07-25 12:02:21,584 [myid:] - INFO  [main:ZKTestCase$1@55] - STARTING testAsyncCreateFailure_NoNode
    [junit] 2016-07-25 12:02:21,584 [myid:] - INFO  [main:ClientBase@448] - Initial fdcount is: 61
    [junit] Running org.apache.zookeeper.test.NettyNettySuiteTest in thread 7
    [junit] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 sec, Thread: 7, Class: org.apache.zookeeper.test.NettyNettySuiteTest
    [junit] Test org.apache.zookeeper.test.NettyNettySuiteTest FAILED (timeout)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build.xml:1207: Tests failed!

Total time: 18 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###################################################################################
############################## FAILED TESTS (if any) ##############################
24 tests failed.
FAILED:  org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testCurrentObserverIsParticipantInNewConfig

Error Message:
client could not connect to reestablished quorum: giving up after 30+ seconds.

Stack Trace:
junit.framework.AssertionFailedError: client could not connect to reestablished quorum: giving up after 30+ seconds.
	at org.apache.zookeeper.test.ReconfigTest.testNormalOperation(ReconfigTest.java:173)
	at org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testCurrentObserverIsParticipantInNewConfig(ReconfigRecoveryTest.java:536)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testLaggingFollowerResyncsUnderNewEpoch

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory.java:754)
	at org.apache.zookeeper.server.quorum.QuorumPeer.startServerCnxnFactory(QuorumPeer.java:1588)
	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:751)
	at org.apache.zookeeper.test.QuorumUtil.start(QuorumUtil.java:211)
	at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testLaggingFollowerResyncsUnderNewEpoch(FollowerResyncConcurrencyTest.java:103)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncByDiffAfterFollowerCrashes

Error Message:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.lang.RuntimeException: java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:122)
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:127)
	at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncByDiffAfterFollowerCrashes(FollowerResyncConcurrencyTest.java:394)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Caused by: java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:90)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testFollowerSendsLastZxid

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory.java:746)
	at org.apache.zookeeper.server.quorum.QuorumPeer.startServerCnxnFactory(QuorumPeer.java:1588)
	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:751)
	at org.apache.zookeeper.test.QuorumUtil.start(QuorumUtil.java:211)
	at org.apache.zookeeper.test.QuorumUtil.startAll(QuorumUtil.java:145)
	at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testFollowerSendsLastZxid(FollowerResyncConcurrencyTest.java:672)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncBySnapThenDiffAfterFollowerCrashes

Error Message:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.lang.RuntimeException: java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:122)
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:127)
	at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.followerResyncCrashTest(FollowerResyncConcurrencyTest.java:199)
	at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncBySnapThenDiffAfterFollowerCrashes(FollowerResyncConcurrencyTest.java:176)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Caused by: java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:90)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testFollowerWatcherResync

Error Message:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.lang.RuntimeException: java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:122)
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:127)
	at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testFollowerWatcherResync(FollowerResyncConcurrencyTest.java:713)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Caused by: java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:90)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.LoadFromLogTest.testLoadFailure

Error Message:
KeeperErrorCode = ConnectionLoss for /data-

Stack Trace:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /data-
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1412)
	at org.apache.zookeeper.test.LoadFromLogTest.testLoadFailure(LoadFromLogTest.java:157)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.NettyNettySuiteTest.testAsyncCreateFailure_NoNode

Error Message:
Timeout occurred. Please note the time in the report does not reflect the time until the timeout.

Stack Trace:
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testHammer

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.quorum.QuorumPeer.<init>(QuorumPeer.java:863)
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:175)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.AsyncHammerTest.setUp(AsyncHammerTest.java:52)
	at org.apache.zookeeper.test.AsyncHammerTest.testHammer(AsyncHammerTest.java:171)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:85)
	at org.apache.zookeeper.test.AsyncHammerTest.setUp(AsyncHammerTest.java:52)
	at org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer(AsyncHammerTest.java:203)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncSetData

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.server.ZooKeeperServer.startSessionTracker(ZooKeeperServer.java:486)
	at org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:444)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.startup(NettyServerCnxnFactory.java:511)
	at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
	at org.apache.zookeeper.test.ClientBase.startServerInstance(ClientBase.java:363)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:469)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncSetDataFailure_NoNode

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.<init>(ClientCommunicatorAdmin.java:45)
	at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.<init>(RMIConnector.java:1467)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:339)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncSetACLFailure_NoNode

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncDeleteFailure_NotEmpty

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.AsyncOpsTest.testAsyncGetChildrenSingle

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.AsyncOpsTest.setUp(AsyncOpsTest.java:48)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.ObserverHierarchicalQuorumTest.testHierarchicalQuorum

Error Message:
KeeperErrorCode = ConnectionLoss for /test-3

Stack Trace:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /test-3
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1412)
	at org.apache.zookeeper.test.ClientHammerTest.runHammer(ClientHammerTest.java:132)
	at org.apache.zookeeper.test.ObserverHierarchicalQuorumTest.testHierarchicalQuorum(ObserverHierarchicalQuorumTest.java:37)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.ObserverQuorumHammerTest.testHammerBasic

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory.java:746)
	at org.apache.zookeeper.server.quorum.QuorumPeer.startServerCnxnFactory(QuorumPeer.java:1588)
	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:751)
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:218)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.ObserverQuorumHammerTest.setUp(ObserverQuorumHammerTest.java:30)


FAILED:  org.apache.zookeeper.test.ObserverTest.testObserver

Error Message:
waiting for server 1 being up

Stack Trace:
junit.framework.AssertionFailedError: waiting for server 1 being up
	at org.apache.zookeeper.test.ObserverTest.testObserver(ObserverTest.java:88)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.QuorumHammerTest.testHammerBasic

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory.java:751)
	at org.apache.zookeeper.server.quorum.QuorumPeer.startServerCnxnFactory(QuorumPeer.java:1588)
	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:751)
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:218)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:78)
	at org.apache.zookeeper.test.QuorumHammerTest.setUp(QuorumHammerTest.java:36)


FAILED:  org.apache.zookeeper.test.QuorumMajorityTest.testMajQuorums

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory.java:746)
	at org.apache.zookeeper.server.quorum.QuorumPeer.startServerCnxnFactory(QuorumPeer.java:1588)
	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:751)
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:216)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:78)


FAILED:  org.apache.zookeeper.test.QuorumMajorityTest.testNull

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory.java:754)
	at org.apache.zookeeper.server.quorum.QuorumPeer.startServerCnxnFactory(QuorumPeer.java:1588)
	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:751)
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:218)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:78)


FAILED:  org.apache.zookeeper.test.QuorumQuotaTest.testQuotaWithQuorum

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory.java:754)
	at org.apache.zookeeper.server.quorum.QuorumPeer.startServerCnxnFactory(QuorumPeer.java:1588)
	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:751)
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:218)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:78)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer(AsyncHammerTest.java:208)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testHammer

Error Message:
null

Stack Trace:
junit.framework.AssertionFailedError
	at org.apache.zookeeper.test.AsyncHammerTest.testHammer(AsyncHammerTest.java:186)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)



ZooKeeper-trunk-jdk8 - Build # 678 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/678/

###################################################################################
########################## LAST 60 LINES OF THE CONSOLE ###########################
[...truncated 434364 lines...]
    [junit] 2016-07-24 11:56:45,874 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10175f91b970000
    [junit] 2016-07-24 11:56:45,875 [myid:] - INFO  [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:19545
    [junit] 2016-07-24 11:56:45,881 [myid:] - INFO  [main:ZooKeeperServer@498] - shutting down
    [junit] 2016-07-24 11:56:45,881 [myid:] - INFO  [main:SessionTrackerImpl@232] - Shutting down
    [junit] 2016-07-24 11:56:45,882 [myid:] - INFO  [main:PrepRequestProcessor@965] - Shutting down
    [junit] 2016-07-24 11:56:45,882 [myid:] - INFO  [main:SyncRequestProcessor@191] - Shutting down
    [junit] 2016-07-24 11:56:45,882 [myid:] - INFO  [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
    [junit] 2016-07-24 11:56:45,882 [myid:] - INFO  [ProcessThread(sid:0 cport:19545)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
    [junit] 2016-07-24 11:56:45,883 [myid:] - INFO  [main:FinalRequestProcessor@479] - shutdown of request processor complete
    [junit] 2016-07-24 11:56:45,883 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port19545,name1=InMemoryDataTree]
    [junit] 2016-07-24 11:56:45,883 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port19545]
    [junit] 2016-07-24 11:56:45,883 [myid:] - INFO  [main:FourLetterWordMain@85] - connecting to 127.0.0.1 19545
    [junit] 2016-07-24 11:56:45,884 [myid:] - INFO  [main:JMXEnv@146] - ensureOnly:[]
    [junit] 2016-07-24 11:56:45,889 [myid:] - INFO  [main:ClientBase@568] - fdcount after test is: 5390 at start it was 5390
    [junit] 2016-07-24 11:56:45,889 [myid:] - INFO  [main:ZKTestCase$1@65] - SUCCEEDED testWatcherAutoResetWithLocal
    [junit] 2016-07-24 11:56:45,889 [myid:] - INFO  [main:ZKTestCase$1@60] - FINISHED testWatcherAutoResetWithLocal
    [junit] Tests run: 101, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 487.276 sec, Thread: 4, Class: org.apache.zookeeper.test.NettyNettySuiteTest
    [junit] 2016-07-24 11:56:46,023 [myid:127.0.0.1:19523] - INFO  [main-SendThread(127.0.0.1:19523):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:19523. Will not attempt to authenticate using SASL (unknown error)
    [junit] 2016-07-24 11:56:46,025 [myid:] - INFO  [New I/O boss #12639:ClientCnxnSocketNetty$1@127] - future isn't success, cause: {}
    [junit] java.net.ConnectException: Connection refused: 127.0.0.1/127.0.0.1:19523
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    [junit] 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    [junit] 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    [junit] 	at java.lang.Thread.run(Thread.java:744)
    [junit] 2016-07-24 11:56:46,025 [myid:] - WARN  [New I/O boss #12639:ClientCnxnSocketNetty$ZKClientHandler@439] - Exception caught: [id: 0xba2ae293] EXCEPTION: java.net.ConnectException: Connection refused: 127.0.0.1/127.0.0.1:19523
    [junit] java.net.ConnectException: Connection refused: 127.0.0.1/127.0.0.1:19523
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
    [junit] 	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    [junit] 	at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
    [junit] 	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    [junit] 	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    [junit] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    [junit] 	at java.lang.Thread.run(Thread.java:744)
    [junit] 2016-07-24 11:56:46,025 [myid:] - INFO  [New I/O boss #12639:ClientCnxnSocketNetty@208] - channel is told closing
    [junit] 2016-07-24 11:56:46,026 [myid:127.0.0.1:19523] - INFO  [main-SendThread(127.0.0.1:19523):ClientCnxn$SendThread@1231] - channel for sessionid 0x0 is lost, closing socket connection and attempting reconnect
    [junit] Test org.apache.zookeeper.test.NettyNettySuiteTest FAILED

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build.xml:1207: Tests failed!

Total time: 12 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###################################################################################
############################## FAILED TESTS (if any) ##############################
18 tests failed.
FAILED:  org.apache.zookeeper.test.ReconfigTest.testRemoveOneAsynchronous

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.quorum.QuorumPeer.<init>(QuorumPeer.java:863)
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:117)
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:127)
	at org.apache.zookeeper.test.ReconfigTest.testRemoveOneAsynchronous(ReconfigTest.java:434)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.SessionTest.testSessionTimeout

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.SessionTest.testMinMaxSessionTimeout

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.SessionTest.testSessionMove

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.SessionTest.testSessionTimeoutAccess

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.SessionTest.testSession

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.SessionTest.testSessionStateNoDupStateReporting

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatchAutoResetWithPending

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherCorrectness

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ClientCommunicatorAdmin.<init>(ClientCommunicatorAdmin.java:45)
	at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.<init>(RMIConnector.java:1467)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:339)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetDisabledWithLocal

Error Message:
Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread

Stack Trace:
java.rmi.ServerError: Error occurred in server thread; nested exception is: 
	java.lang.OutOfMemoryError: unable to create new native thread
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:351)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:276)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:253)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
	at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2404)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:229)
	at org.apache.zookeeper.test.JMXEnv.setUp(JMXEnv.java:58)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:454)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin.<init>(ServerCommunicatorAdmin.java:50)
	at javax.management.remote.rmi.RMIConnectionImpl$RMIServerCommunicatorAdmin.<init>(RMIConnectionImpl.java:1319)
	at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:160)
	at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:173)
	at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:250)
	at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:199)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
	at sun.rmi.transport.Transport$1.run(Transport.java:178)
	at sun.rmi.transport.Transport$1.run(Transport.java:175)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:174)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:557)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:812)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:671)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetWithGlobal

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherCount

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetDisabledWithGlobal

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetWithLocal

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.quorum.QuorumPeer.<init>(QuorumPeer.java:863)
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:172)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.AsyncHammerTest.setUp(AsyncHammerTest.java:52)
	at org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer(AsyncHammerTest.java:203)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.ReconfigTest.testRemoveOneAsynchronous

Error Message:
Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNetty

Stack Trace:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNetty
	at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:2936)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:870)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:788)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:928)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:675)
	at org.apache.zookeeper.test.ReconfigTest.createHandles(ReconfigTest.java:203)
	at org.apache.zookeeper.test.ReconfigTest.testRemoveOneAsynchronous(ReconfigTest.java:437)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Caused by: java.lang.reflect.InvocationTargetException
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:2933)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
	at org.apache.zookeeper.ClientCnxnSocketNetty.<init>(ClientCnxnSocketNetty.java:72)


FAILED:  org.apache.zookeeper.test.ReconfigTest.testBulkReconfig

Error Message:
waiting for server up

Stack Trace:
junit.framework.AssertionFailedError: waiting for server up
	at org.apache.zookeeper.test.QuorumUtil.startAll(QuorumUtil.java:151)
	at org.apache.zookeeper.test.ReconfigTest.testBulkReconfig(ReconfigTest.java:398)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testHammer

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.test.AsyncHammerTest.testHammer(AsyncHammerTest.java:177)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)



ZooKeeper-trunk-jdk8 - Build # 677 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/677/

###################################################################################
########################## LAST 60 LINES OF THE CONSOLE ###########################
[...truncated 436229 lines...]
    [junit] 	at org.apache.zookeeper.server.NettyServerCnxn.sendCloseSession(NettyServerCnxn.java:441)
    [junit] 	at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:459)
    [junit] 	at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:182)
    [junit] 	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:113)
    [junit] 2016-07-23 11:57:06,624 [myid:] - INFO  [main:ZooKeeper@1313] - Session: 0x10170d30ade0000 closed
    [junit] 2016-07-23 11:57:06,624 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 167199
    [junit] 2016-07-23 11:57:06,625 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 1642
    [junit] 2016-07-23 11:57:06,625 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testWatcherAutoResetWithLocal
    [junit] 2016-07-23 11:57:06,626 [myid:] - INFO  [main:ClientBase@543] - tearDown starting
    [junit] 2016-07-23 11:57:06,626 [myid:] - INFO  [main:ClientBase@513] - STOPPING server
    [junit] 2016-07-23 11:57:06,627 [myid:] - INFO  [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:27614
    [junit] 2016-07-23 11:57:06,624 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10170d30ade0000
    [junit] 2016-07-23 11:57:06,631 [myid:] - INFO  [main:ZooKeeperServer@498] - shutting down
    [junit] 2016-07-23 11:57:06,631 [myid:] - INFO  [main:SessionTrackerImpl@232] - Shutting down
    [junit] 2016-07-23 11:57:06,632 [myid:] - INFO  [main:PrepRequestProcessor@965] - Shutting down
    [junit] 2016-07-23 11:57:06,632 [myid:] - INFO  [main:SyncRequestProcessor@191] - Shutting down
    [junit] 2016-07-23 11:57:06,632 [myid:] - INFO  [ProcessThread(sid:0 cport:27614)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
    [junit] 2016-07-23 11:57:06,633 [myid:] - INFO  [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
    [junit] 2016-07-23 11:57:06,633 [myid:] - INFO  [main:FinalRequestProcessor@479] - shutdown of request processor complete
    [junit] 2016-07-23 11:57:06,633 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port27614,name1=InMemoryDataTree]
    [junit] 2016-07-23 11:57:06,634 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port27614]
    [junit] 2016-07-23 11:57:06,634 [myid:] - INFO  [main:FourLetterWordMain@85] - connecting to 127.0.0.1 27614
    [junit] 2016-07-23 11:57:06,634 [myid:] - INFO  [main:JMXEnv@146] - ensureOnly:[]
    [junit] 2016-07-23 11:57:06,638 [myid:] - INFO  [main:ClientBase@568] - fdcount after test is: 4832 at start it was 4832
    [junit] 2016-07-23 11:57:06,638 [myid:] - INFO  [main:ZKTestCase$1@65] - SUCCEEDED testWatcherAutoResetWithLocal
    [junit] 2016-07-23 11:57:06,638 [myid:] - INFO  [main:ZKTestCase$1@60] - FINISHED testWatcherAutoResetWithLocal
    [junit] 2016-07-23 11:57:06,666 [myid:127.0.0.1:27504] - INFO  [main-SendThread(127.0.0.1:27504):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:27504. Will not attempt to authenticate using SASL (unknown error)
    [junit] 2016-07-23 11:57:06,667 [myid:127.0.0.1:27504] - WARN  [main-SendThread(127.0.0.1:27504):ClientCnxn$SendThread@1235] - Session 0x20170d0371c0000 for server 127.0.0.1/127.0.0.1:27504, unexpected error, closing socket connection and attempting reconnect
    [junit] java.net.ConnectException: Connection refused
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
    [junit] 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
    [junit] Tests run: 101, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 487.8 sec, Thread: 7, Class: org.apache.zookeeper.test.NioNettySuiteTest
    [junit] 2016-07-23 11:57:06,713 [myid:127.0.0.1:27501] - INFO  [main-SendThread(127.0.0.1:27501):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:27501. Will not attempt to authenticate using SASL (unknown error)
    [junit] 2016-07-23 11:57:06,713 [myid:127.0.0.1:27501] - WARN  [main-SendThread(127.0.0.1:27501):ClientCnxn$SendThread@1235] - Session 0x10170d0371c0000 for server 127.0.0.1/127.0.0.1:27501, unexpected error, closing socket connection and attempting reconnect
    [junit] java.net.ConnectException: Connection refused
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
    [junit] 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
    [junit] 2016-07-23 11:57:06,805 [myid:127.0.0.1:27507] - INFO  [main-SendThread(127.0.0.1:27507):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:27507. Will not attempt to authenticate using SASL (unknown error)
    [junit] 2016-07-23 11:57:06,805 [myid:127.0.0.1:27507] - WARN  [main-SendThread(127.0.0.1:27507):ClientCnxn$SendThread@1235] - Session 0x30170d0378e0000 for server 127.0.0.1/127.0.0.1:27507, unexpected error, closing socket connection and attempting reconnect
    [junit] java.net.ConnectException: Connection refused
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
    [junit] 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
    [junit] Test org.apache.zookeeper.test.NioNettySuiteTest FAILED

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build.xml:1207: Tests failed!

Total time: 12 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###################################################################################
############################## FAILED TESTS (if any) ##############################
20 tests failed.
FAILED:  org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testCurrentServersAreObserversInNextConfig

Error Message:
waiting for server 0 being up

Stack Trace:
junit.framework.AssertionFailedError: waiting for server 0 being up
	at org.apache.zookeeper.server.quorum.ReconfigRecoveryTest.testCurrentServersAreObserversInNextConfig(ReconfigRecoveryTest.java:217)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.LoadFromLogTest.testLoadFailure

Error Message:
KeeperErrorCode = ConnectionLoss for /data-

Stack Trace:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /data-
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1412)
	at org.apache.zookeeper.test.LoadFromLogTest.testLoadFailure(LoadFromLogTest.java:157)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.quorum.QuorumPeer.<init>(QuorumPeer.java:863)
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:181)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.AsyncHammerTest.setUp(AsyncHammerTest.java:52)
	at org.apache.zookeeper.test.AsyncHammerTest.testObserversHammer(AsyncHammerTest.java:203)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.ReconfigTest.testJMXBeanAfterRoleChange

Error Message:
Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNetty

Stack Trace:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNetty
	at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:2936)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:870)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:788)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:928)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:675)
	at org.apache.zookeeper.test.ReconfigTest.createHandles(ReconfigTest.java:203)
	at org.apache.zookeeper.test.ReconfigTest.testJMXBeanAfterRoleChange(ReconfigTest.java:928)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Caused by: java.lang.reflect.InvocationTargetException
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:2933)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
	at org.apache.zookeeper.ClientCnxnSocketNetty.<init>(ClientCnxnSocketNetty.java:72)


FAILED:  org.apache.zookeeper.test.ReconfigTest.testRemoveOneAsynchronous

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.quorum.QuorumPeer.<init>(QuorumPeer.java:863)
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:117)
	at org.apache.zookeeper.test.QuorumUtil.<init>(QuorumUtil.java:127)
	at org.apache.zookeeper.test.ReconfigTest.testRemoveOneAsynchronous(ReconfigTest.java:434)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.SessionTest.testSessionTimeout

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.SessionTest.testMinMaxSessionTimeout

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.SessionTest.testSessionMove

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.SessionTest.testSessionTimeoutAccess

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.SessionTest.testSession

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.SessionTest.testSessionStateNoDupStateReporting

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.SessionTest.setUp(SessionTest.java:77)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherCorrectness

Error Message:
Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNetty

Stack Trace:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNetty
	at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:2936)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:870)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:788)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:928)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:675)
	at org.apache.zookeeper.TestableZooKeeper.<init>(TestableZooKeeper.java:35)
	at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:185)
	at org.apache.zookeeper.test.ClientBase.createClient(ClientBase.java:177)
	at org.apache.zookeeper.test.WatcherTest.testWatcherCorrectness(WatcherTest.java:96)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Caused by: java.lang.reflect.InvocationTargetException
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:2933)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:151)
	at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.<init>(NioClientSocketChannelFactory.java:116)
	at org.apache.zookeeper.ClientCnxnSocketNetty.<init>(ClientCnxnSocketNetty.java:72)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetDisabledWithLocal

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.server.ZooKeeperServer.startSessionTracker(ZooKeeperServer.java:486)
	at org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:444)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.startup(NettyServerCnxnFactory.java:511)
	at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
	at org.apache.zookeeper.test.ClientBase.startServerInstance(ClientBase.java:363)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:469)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetWithGlobal

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherCount

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetDisabledWithGlobal

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetWithLocal

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.ReconfigTest.testBulkReconfig

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.ClientCnxn.start(ClientCnxn.java:400)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:871)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:788)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:928)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:675)
	at org.apache.zookeeper.test.ReconfigTest.createHandles(ReconfigTest.java:203)
	at org.apache.zookeeper.test.ReconfigTest.testBulkReconfig(ReconfigTest.java:399)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatchAutoResetWithPending

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368)
	at org.jboss.netty.util.internal.DeadLockProofWorker.start(DeadLockProofWorker.java:38)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:368)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.<init>(AbstractNioSelector.java:100)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.<init>(AbstractNioWorker.java:52)
	at org.jboss.netty.channel.socket.nio.NioWorker.<init>(NioWorker.java:45)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:44)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.newWorker(NioWorkerPool.java:28)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:80)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:39)
	at org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(NioWorkerPool.java:33)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:149)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:131)
	at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.<init>(NioServerSocketChannelFactory.java:115)
	at org.apache.zookeeper.server.NettyServerCnxnFactory.<init>(NettyServerCnxnFactory.java:332)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at java.lang.Class.newInstance(Class.java:433)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:134)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:152)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.ClientBase.createNewServerInstance(ClientBase.java:388)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:467)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.WatcherTest.setUp(WatcherTest.java:73)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testHammer

Error Message:
unable to create new native thread

Stack Trace:
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:713)
	at org.apache.zookeeper.test.AsyncHammerTest.testHammer(AsyncHammerTest.java:177)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)



ZooKeeper-trunk-jdk8 - Build # 676 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/676/

###################################################################################
########################## LAST 60 LINES OF THE CONSOLE ###########################
[...truncated 440296 lines...]
    [junit] 2016-07-22 12:58:57,202 [myid:] - INFO  [main:JMXEnv@146] - ensureOnly:[]
    [junit] 2016-07-22 12:58:57,204 [myid:] - INFO  [main:ClientBase@466] - STARTING server
    [junit] 2016-07-22 12:58:57,204 [myid:] - INFO  [main:ClientBase@386] - CREATING server instance 127.0.0.1:11222
    [junit] 2016-07-22 12:58:57,205 [myid:] - INFO  [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 3 selector thread(s), 48 worker threads, and 64 kB direct buffers.
    [junit] 2016-07-22 12:58:57,206 [myid:] - INFO  [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
    [junit] 2016-07-22 12:58:57,206 [myid:] - INFO  [main:ClientBase@361] - STARTING server instance 127.0.0.1:11222
    [junit] 2016-07-22 12:58:57,207 [myid:] - INFO  [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
    [junit] 2016-07-22 12:58:57,207 [myid:] - INFO  [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
    [junit] 2016-07-22 12:58:57,207 [myid:] - INFO  [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /x1/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build/test/tmp/test747260990386060268.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build/test/tmp/test747260990386060268.junit.dir/version-2
    [junit] 2016-07-22 12:58:57,211 [myid:] - INFO  [main:FileSnap@83] - Reading snapshot /x1/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build/test/tmp/test747260990386060268.junit.dir/version-2/snapshot.b
    [junit] 2016-07-22 12:58:57,215 [myid:] - INFO  [main:FileTxnSnapLog@298] - Snapshotting: 0xb to /x1/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build/test/tmp/test747260990386060268.junit.dir/version-2/snapshot.b
    [junit] 2016-07-22 12:58:57,218 [myid:] - INFO  [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
    [junit] 2016-07-22 12:58:57,219 [myid:] - INFO  [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:42585
    [junit] 2016-07-22 12:58:57,220 [myid:] - INFO  [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:42585
    [junit] 2016-07-22 12:58:57,220 [myid:] - INFO  [NIOWorkerThread-1:StatCommand@49] - Stat command output
    [junit] 2016-07-22 12:58:57,221 [myid:] - INFO  [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:42585 (no session established for client)
    [junit] 2016-07-22 12:58:57,221 [myid:] - INFO  [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
    [junit] 2016-07-22 12:58:57,223 [myid:] - INFO  [main:JMXEnv@245] - expect:InMemoryDataTree
    [junit] 2016-07-22 12:58:57,224 [myid:] - INFO  [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree
    [junit] 2016-07-22 12:58:57,224 [myid:] - INFO  [main:JMXEnv@245] - expect:StandaloneServer_port
    [junit] 2016-07-22 12:58:57,224 [myid:] - INFO  [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11222
    [junit] 2016-07-22 12:58:57,225 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 5730
    [junit] 2016-07-22 12:58:57,225 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 25
    [junit] 2016-07-22 12:58:57,225 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testQuota
    [junit] 2016-07-22 12:58:57,226 [myid:] - INFO  [main:ClientBase@543] - tearDown starting
    [junit] 2016-07-22 12:58:57,277 [myid:] - INFO  [main:ZooKeeper@1313] - Session: 0x10081ae96ae0000 closed
    [junit] 2016-07-22 12:58:57,277 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10081ae96ae0000
    [junit] 2016-07-22 12:58:57,278 [myid:] - INFO  [main:ClientBase@513] - STOPPING server
    [junit] 2016-07-22 12:58:57,278 [myid:] - INFO  [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
    [junit] 2016-07-22 12:58:57,279 [myid:] - INFO  [NIOServerCxnFactory.SelectorThread-2:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
    [junit] 2016-07-22 12:58:57,278 [myid:] - INFO  [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
    [junit] 2016-07-22 12:58:57,279 [myid:] - INFO  [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
    [junit] 2016-07-22 12:58:57,282 [myid:] - INFO  [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
    [junit] 2016-07-22 12:58:57,283 [myid:] - INFO  [main:ZooKeeperServer@498] - shutting down
    [junit] 2016-07-22 12:58:57,283 [myid:] - INFO  [main:SessionTrackerImpl@232] - Shutting down
    [junit] 2016-07-22 12:58:57,283 [myid:] - INFO  [main:PrepRequestProcessor@965] - Shutting down
    [junit] 2016-07-22 12:58:57,284 [myid:] - INFO  [main:SyncRequestProcessor@191] - Shutting down
    [junit] 2016-07-22 12:58:57,284 [myid:] - INFO  [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
    [junit] 2016-07-22 12:58:57,284 [myid:] - INFO  [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
    [junit] 2016-07-22 12:58:57,284 [myid:] - INFO  [main:FinalRequestProcessor@479] - shutdown of request processor complete
    [junit] 2016-07-22 12:58:57,285 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree]
    [junit] 2016-07-22 12:58:57,285 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222]
    [junit] 2016-07-22 12:58:57,286 [myid:] - INFO  [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
    [junit] 2016-07-22 12:58:57,287 [myid:] - INFO  [main:JMXEnv@146] - ensureOnly:[]
    [junit] 2016-07-22 12:58:57,293 [myid:] - INFO  [main:ClientBase@568] - fdcount after test is: 52 at start it was 37
    [junit] 2016-07-22 12:58:57,294 [myid:] - INFO  [main:ClientBase@570] - sleeping for 20 secs
    [junit] 2016-07-22 12:58:57,295 [myid:] - INFO  [main:ZKTestCase$1@65] - SUCCEEDED testQuota
    [junit] 2016-07-22 12:58:57,295 [myid:] - INFO  [main:ZKTestCase$1@60] - FINISHED testQuota
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.804 sec

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build.xml:1233: Tests failed!

Total time: 74 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###################################################################################
############################## FAILED TESTS (if any) ##############################
2 tests failed.
FAILED:  org.apache.zookeeper.test.WatcherTest.testWatchAutoResetWithPending

Error Message:
For 50 expected:<1> but was:<0>

Stack Trace:
junit.framework.AssertionFailedError: For 50 expected:<1> but was:<0>
	at org.apache.zookeeper.test.WatcherTest.testWatchAutoResetWithPending(WatcherTest.java:215)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testHammer

Error Message:
null

Stack Trace:
junit.framework.AssertionFailedError
	at org.apache.zookeeper.test.AsyncHammerTest.testHammer(AsyncHammerTest.java:186)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)



ZooKeeper-trunk-jdk8 - Build # 675 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/675/

###################################################################################
########################## LAST 60 LINES OF THE CONSOLE ###########################
[...truncated 394602 lines...]
    [junit] 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
    [junit] 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
    [junit] 2016-07-21 14:05:10,499 [myid:127.0.0.1:27624] - INFO  [main-SendThread(127.0.0.1:27624):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:27624. Will not attempt to authenticate using SASL (unknown error)
    [junit] 2016-07-21 14:05:10,499 [myid:127.0.0.1:27624] - INFO  [main-SendThread(127.0.0.1:27624):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /127.0.0.1:52180, server: 127.0.0.1/127.0.0.1:27624
    [junit] 2016-07-21 14:05:10,504 [myid:] - INFO  [New I/O worker #6635:ZooKeeperServer@969] - Client attempting to renew session 0x101676954c10000 at /127.0.0.1:52180
    [junit] 2016-07-21 14:05:10,510 [myid:] - INFO  [New I/O worker #6635:ZooKeeperServer@678] - Established session 0x101676954c10000 with negotiated timeout 6000 for client /127.0.0.1:52180
    [junit] 2016-07-21 14:05:10,510 [myid:127.0.0.1:27624] - INFO  [main-SendThread(127.0.0.1:27624):ClientCnxn$SendThread@1381] - Session establishment complete on server 127.0.0.1/127.0.0.1:27624, sessionid = 0x101676954c10000, negotiated timeout = 6000
    [junit] 2016-07-21 14:05:10,521 [myid:] - INFO  [SyncThread:0:FileTxnLog@204] - Creating new log file: log.7
    [junit] 2016-07-21 14:05:10,587 [myid:] - INFO  [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
    [junit] 2016-07-21 14:05:10,587 [myid:] - INFO  [SessionTracker:SessionTrackerImpl@158] - SessionTrackerImpl exited loop!
    [junit] 2016-07-21 14:05:10,705 [myid:127.0.0.1:27507] - INFO  [main-SendThread(127.0.0.1:27507):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:27507. Will not attempt to authenticate using SASL (unknown error)
    [junit] 2016-07-21 14:05:10,705 [myid:127.0.0.1:27507] - WARN  [main-SendThread(127.0.0.1:27507):ClientCnxn$SendThread@1235] - Session 0x30167645b2f0000 for server 127.0.0.1/127.0.0.1:27507, unexpected error, closing socket connection and attempting reconnect
    [junit] java.net.ConnectException: Connection refused
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
    [junit] 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
    [junit] 2016-07-21 14:05:11,571 [myid:] - INFO  [ProcessThread(sid:0 cport:27624)::PrepRequestProcessor@647] - Processed session termination for sessionid: 0x101676954c10000
    [junit] 2016-07-21 14:05:11,584 [myid:] - INFO  [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port27624,name1=Connections,name2=127.0.0.1,name3=0x101676954c10000]
    [junit] 2016-07-21 14:05:11,584 [myid:] - INFO  [main:ZooKeeper@1313] - Session: 0x101676954c10000 closed
    [junit] 2016-07-21 14:05:11,585 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x101676954c10000
    [junit] 2016-07-21 14:05:11,585 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 142317
    [junit] 2016-07-21 14:05:11,585 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 1641
    [junit] 2016-07-21 14:05:11,585 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testWatcherAutoResetWithLocal
    [junit] 2016-07-21 14:05:11,585 [myid:] - INFO  [main:ClientBase@543] - tearDown starting
    [junit] 2016-07-21 14:05:11,585 [myid:] - INFO  [main:ClientBase@513] - STOPPING server
    [junit] 2016-07-21 14:05:11,585 [myid:] - INFO  [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:27624
    [junit] 2016-07-21 14:05:11,590 [myid:] - INFO  [main:ZooKeeperServer@498] - shutting down
    [junit] 2016-07-21 14:05:11,590 [myid:] - INFO  [main:SessionTrackerImpl@232] - Shutting down
    [junit] 2016-07-21 14:05:11,590 [myid:] - INFO  [main:PrepRequestProcessor@965] - Shutting down
    [junit] 2016-07-21 14:05:11,590 [myid:] - INFO  [main:SyncRequestProcessor@191] - Shutting down
    [junit] 2016-07-21 14:05:11,594 [myid:] - INFO  [ProcessThread(sid:0 cport:27624)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
    [junit] 2016-07-21 14:05:11,594 [myid:] - INFO  [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
    [junit] 2016-07-21 14:05:11,602 [myid:] - INFO  [main:FinalRequestProcessor@479] - shutdown of request processor complete
    [junit] 2016-07-21 14:05:11,602 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port27624,name1=InMemoryDataTree]
    [junit] 2016-07-21 14:05:11,602 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port27624]
    [junit] 2016-07-21 14:05:11,633 [myid:] - INFO  [main:FourLetterWordMain@85] - connecting to 127.0.0.1 27624
    [junit] 2016-07-21 14:05:11,634 [myid:] - INFO  [main:JMXEnv@146] - ensureOnly:[]
    [junit] 2016-07-21 14:05:11,644 [myid:] - INFO  [main:ClientBase@568] - fdcount after test is: 4820 at start it was 4820
    [junit] 2016-07-21 14:05:11,645 [myid:] - INFO  [main:ZKTestCase$1@65] - SUCCEEDED testWatcherAutoResetWithLocal
    [junit] 2016-07-21 14:05:11,645 [myid:] - INFO  [main:ZKTestCase$1@60] - FINISHED testWatcherAutoResetWithLocal
    [junit] Tests run: 101, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 754.936 sec, Thread: 7, Class: org.apache.zookeeper.test.NioNettySuiteTest
    [junit] 2016-07-21 14:05:11,764 [myid:127.0.0.1:27501] - INFO  [main-SendThread(127.0.0.1:27501):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:27501. Will not attempt to authenticate using SASL (unknown error)
    [junit] 2016-07-21 14:05:11,765 [myid:127.0.0.1:27501] - WARN  [main-SendThread(127.0.0.1:27501):ClientCnxn$SendThread@1235] - Session 0x10167645af00000 for server 127.0.0.1/127.0.0.1:27501, unexpected error, closing socket connection and attempting reconnect
    [junit] java.net.ConnectException: Connection refused
    [junit] 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    [junit] 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    [junit] 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
    [junit] 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build.xml:1207: Tests failed!

Total time: 22 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###################################################################################
############################## FAILED TESTS (if any) ##############################
17 tests failed.
FAILED:  org.apache.zookeeper.server.quorum.RaceConditionTest.testRaceConditionBetweenLeaderAndAckRequestProcessor

Error Message:
test timed out after 30000 milliseconds

Stack Trace:
org.junit.runners.model.TestTimedOutException: test timed out after 30000 milliseconds
	at org.apache.log4j.Category.callAppenders(Category.java:204)
	at org.apache.log4j.Category.forcedLog(Category.java:391)
	at org.apache.log4j.Category.log(Category.java:856)
	at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:323)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.server.quorum.Zab1_0Test.testNormalObserverRun

Error Message:
Timeout occurred. Please note the time in the report does not reflect the time until the timeout.

Stack Trace:
junit.framework.AssertionFailedError: Timeout occurred. Please note the time in the report does not reflect the time until the timeout.
	at java.lang.Thread.run(Thread.java:744)


FAILED:  org.apache.zookeeper.test.ClientHammerTest.testHammerSuper

Error Message:
expected:<10> but was:<6>

Stack Trace:
junit.framework.AssertionFailedError: expected:<10> but was:<6>
	at org.apache.zookeeper.test.ClientHammerTest.verifyHammer(ClientHammerTest.java:231)
	at org.apache.zookeeper.test.ClientHammerTest.testHammerSuper(ClientHammerTest.java:181)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncBySnapThenDiffAfterFollowerCrashes

Error Message:
Waiting for server up

Stack Trace:
junit.framework.AssertionFailedError: Waiting for server up
	at org.apache.zookeeper.test.QuorumUtil.restart(QuorumUtil.java:216)
	at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.followerResyncCrashTest(FollowerResyncConcurrencyTest.java:237)
	at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testResyncBySnapThenDiffAfterFollowerCrashes(FollowerResyncConcurrencyTest.java:176)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testFollowerWatcherResync

Error Message:
waiting for server up

Stack Trace:
junit.framework.AssertionFailedError: waiting for server up
	at org.apache.zookeeper.test.QuorumUtil.startAll(QuorumUtil.java:151)
	at org.apache.zookeeper.test.FollowerResyncConcurrencyTest.testFollowerWatcherResync(FollowerResyncConcurrencyTest.java:714)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.FourLetterWordsQuorumTest.testFourLetterWords

Error Message:
Exception while executing four letter word: ruok

Stack Trace:
java.io.IOException: Exception while executing four letter word: ruok
	at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:124)
	at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:54)
	at org.apache.zookeeper.test.FourLetterWordsQuorumTest.verify(FourLetterWordsQuorumTest.java:109)
	at org.apache.zookeeper.test.FourLetterWordsQuorumTest.testFourLetterWords(FourLetterWordsQuorumTest.java:82)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:150)
	at java.net.SocketInputStream.read(SocketInputStream.java:121)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at org.apache.zookeeper.client.FourLetterWordMain.send4LetterWord(FourLetterWordMain.java:119)


FAILED:  org.apache.zookeeper.test.LoadFromLogTest.testRestoreWithTransactionErrors

Error Message:
KeeperErrorCode = ConnectionLoss for /invaliddir/test-

Stack Trace:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /invaliddir/test-
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1412)
	at org.apache.zookeeper.test.LoadFromLogTest.testRestoreWithTransactionErrors(LoadFromLogTest.java:452)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.LoadFromLogTest.testLoad

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:414)
	at sun.nio.ch.Net.bind(Net.java:406)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:687)
	at org.apache.zookeeper.server.ServerCnxnFactory.configure(ServerCnxnFactory.java:76)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:153)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.LoadFromLogTest.testLoad(LoadFromLogTest.java:87)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.LoadFromLogTest.testDatadirAutocreate

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:414)
	at sun.nio.ch.Net.bind(Net.java:406)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:687)
	at org.apache.zookeeper.server.ServerCnxnFactory.configure(ServerCnxnFactory.java:76)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:153)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.LoadFromLogTest.testDatadirAutocreate(LoadFromLogTest.java:495)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.LoadFromLogTest.testRestore

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:414)
	at sun.nio.ch.Net.bind(Net.java:406)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:687)
	at org.apache.zookeeper.server.ServerCnxnFactory.configure(ServerCnxnFactory.java:76)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:153)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.LoadFromLogTest.testRestore(LoadFromLogTest.java:361)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.LoadFromLogTest.testLoadFailure

Error Message:
Address already in use

Stack Trace:
java.net.BindException: Address already in use
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:414)
	at sun.nio.ch.Net.bind(Net.java:406)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:687)
	at org.apache.zookeeper.server.ServerCnxnFactory.configure(ServerCnxnFactory.java:76)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:153)
	at org.apache.zookeeper.server.ServerCnxnFactory.createFactory(ServerCnxnFactory.java:146)
	at org.apache.zookeeper.test.LoadFromLogTest.testLoadFailure(LoadFromLogTest.java:148)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.LocalSessionRequestTest.testLocalSessionsOnLeader

Error Message:
waiting for server up

Stack Trace:
junit.framework.AssertionFailedError: waiting for server up
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:223)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:78)
	at org.apache.zookeeper.test.LocalSessionRequestTest.setUp(LocalSessionRequestTest.java:53)


FAILED:  org.apache.zookeeper.test.LocalSessionsOnlyTest.testLocalSessionsOnLeader

Error Message:
waiting for server up

Stack Trace:
junit.framework.AssertionFailedError: waiting for server up
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:223)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:78)
	at org.apache.zookeeper.test.LocalSessionsOnlyTest.setUp(LocalSessionsOnlyTest.java:55)


FAILED:  org.apache.zookeeper.test.AsyncHammerTest.testHammer

Error Message:
waiting for server up

Stack Trace:
junit.framework.AssertionFailedError: waiting for server up
	at org.apache.zookeeper.test.QuorumBase.startServers(QuorumBase.java:223)
	at org.apache.zookeeper.test.QuorumBase.setUp(QuorumBase.java:120)
	at org.apache.zookeeper.test.AsyncHammerTest.setUp(AsyncHammerTest.java:52)
	at org.apache.zookeeper.test.AsyncHammerTest.testHammer(AsyncHammerTest.java:171)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.ReconfigTest.testBulkReconfig

Error Message:
waiting for server up

Stack Trace:
junit.framework.AssertionFailedError: waiting for server up
	at org.apache.zookeeper.test.QuorumUtil.startAll(QuorumUtil.java:151)
	at org.apache.zookeeper.test.ReconfigTest.testBulkReconfig(ReconfigTest.java:398)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)


FAILED:  org.apache.zookeeper.test.SSLAuthTest.testRejection

Error Message:
waiting for server up

Stack Trace:
junit.framework.AssertionFailedError: waiting for server up
	at org.apache.zookeeper.test.ClientBase.startServerInstance(ClientBase.java:364)
	at org.apache.zookeeper.test.ClientBase.startServer(ClientBase.java:469)
	at org.apache.zookeeper.test.ClientBase.setUp(ClientBase.java:460)
	at org.apache.zookeeper.test.SSLAuthTest.setUp(SSLAuthTest.java:55)


FAILED:  org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetDisabledWithGlobal

Error Message:
Did not connect

Stack Trace:
java.util.concurrent.TimeoutException: Did not connect
	at org.apache.zookeeper.test.ClientBase$CountdownWatcher.waitForConnected(ClientBase.java:132)
	at org.apache.zookeeper.test.WatcherTest.testWatcherAutoReset(WatcherTest.java:374)
	at org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetWithGlobal(WatcherTest.java:231)
	at org.apache.zookeeper.test.WatcherTest.testWatcherAutoResetDisabledWithGlobal(WatcherTest.java:250)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)



ZooKeeper-trunk-jdk8 - Build # 674 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/674/

###################################################################################
########################## LAST 60 LINES OF THE CONSOLE ###########################
[...truncated 426800 lines...]
    [junit] 2016-07-20 13:37:47,896 [myid:] - INFO  [main:JMXEnv@146] - ensureOnly:[]
    [junit] 2016-07-20 13:37:47,897 [myid:] - INFO  [main:ClientBase@466] - STARTING server
    [junit] 2016-07-20 13:37:47,897 [myid:] - INFO  [main:ClientBase@386] - CREATING server instance 127.0.0.1:11222
    [junit] 2016-07-20 13:37:47,898 [myid:] - INFO  [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with 10s sessionless connection timeout, 3 selector thread(s), 48 worker threads, and 64 kB direct buffers.
    [junit] 2016-07-20 13:37:47,898 [myid:] - INFO  [main:NIOServerCnxnFactory@686] - binding to port 0.0.0.0/0.0.0.0:11222
    [junit] 2016-07-20 13:37:47,898 [myid:] - INFO  [main:ClientBase@361] - STARTING server instance 127.0.0.1:11222
    [junit] 2016-07-20 13:37:47,899 [myid:] - INFO  [main:ZooKeeperServer@858] - minSessionTimeout set to 6000
    [junit] 2016-07-20 13:37:47,899 [myid:] - INFO  [main:ZooKeeperServer@867] - maxSessionTimeout set to 60000
    [junit] 2016-07-20 13:37:47,899 [myid:] - INFO  [main:ZooKeeperServer@156] - Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /x1/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build/test/tmp/test4683344467839359790.junit.dir/version-2 snapdir /x1/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build/test/tmp/test4683344467839359790.junit.dir/version-2
    [junit] 2016-07-20 13:37:47,900 [myid:] - INFO  [main:FileSnap@83] - Reading snapshot /x1/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build/test/tmp/test4683344467839359790.junit.dir/version-2/snapshot.b
    [junit] 2016-07-20 13:37:47,902 [myid:] - INFO  [main:FileTxnSnapLog@298] - Snapshotting: 0xb to /x1/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build/test/tmp/test4683344467839359790.junit.dir/version-2/snapshot.b
    [junit] 2016-07-20 13:37:47,904 [myid:] - INFO  [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
    [junit] 2016-07-20 13:37:47,904 [myid:] - INFO  [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:50515
    [junit] 2016-07-20 13:37:47,905 [myid:] - INFO  [NIOWorkerThread-1:NIOServerCnxn@485] - Processing stat command from /127.0.0.1:50515
    [junit] 2016-07-20 13:37:47,905 [myid:] - INFO  [NIOWorkerThread-1:StatCommand@49] - Stat command output
    [junit] 2016-07-20 13:37:47,906 [myid:] - INFO  [NIOWorkerThread-1:NIOServerCnxn@607] - Closed socket connection for client /127.0.0.1:50515 (no session established for client)
    [junit] 2016-07-20 13:37:47,906 [myid:] - INFO  [main:JMXEnv@228] - ensureParent:[InMemoryDataTree, StandaloneServer_port]
    [junit] 2016-07-20 13:37:47,908 [myid:] - INFO  [main:JMXEnv@245] - expect:InMemoryDataTree
    [junit] 2016-07-20 13:37:47,908 [myid:] - INFO  [main:JMXEnv@249] - found:InMemoryDataTree org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree
    [junit] 2016-07-20 13:37:47,908 [myid:] - INFO  [main:JMXEnv@245] - expect:StandaloneServer_port
    [junit] 2016-07-20 13:37:47,908 [myid:] - INFO  [main:JMXEnv@249] - found:StandaloneServer_port org.apache.ZooKeeperService:name0=StandaloneServer_port11222
    [junit] 2016-07-20 13:37:47,908 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 5553
    [junit] 2016-07-20 13:37:47,909 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 25
    [junit] 2016-07-20 13:37:47,909 [myid:] - INFO  [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testQuota
    [junit] 2016-07-20 13:37:47,909 [myid:] - INFO  [main:ClientBase@543] - tearDown starting
    [junit] 2016-07-20 13:37:47,968 [myid:] - INFO  [main:ZooKeeper@1313] - Session: 0x10077856f180000 closed
    [junit] 2016-07-20 13:37:47,968 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x10077856f180000
    [junit] 2016-07-20 13:37:47,968 [myid:] - INFO  [main:ClientBase@513] - STOPPING server
    [junit] 2016-07-20 13:37:47,969 [myid:] - INFO  [ConnnectionExpirer:NIOServerCnxnFactory$ConnectionExpirerThread@583] - ConnnectionExpirerThread interrupted
    [junit] 2016-07-20 13:37:47,969 [myid:] - INFO  [NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:11222:NIOServerCnxnFactory$AcceptThread@219] - accept thread exitted run method
    [junit] 2016-07-20 13:37:47,969 [myid:] - INFO  [NIOServerCxnFactory.SelectorThread-1:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
    [junit] 2016-07-20 13:37:47,969 [myid:] - INFO  [NIOServerCxnFactory.SelectorThread-2:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
    [junit] 2016-07-20 13:37:47,969 [myid:] - INFO  [NIOServerCxnFactory.SelectorThread-0:NIOServerCnxnFactory$SelectorThread@420] - selector thread exitted run method
    [junit] 2016-07-20 13:37:47,970 [myid:] - INFO  [main:ZooKeeperServer@498] - shutting down
    [junit] 2016-07-20 13:37:47,970 [myid:] - INFO  [main:SessionTrackerImpl@232] - Shutting down
    [junit] 2016-07-20 13:37:47,970 [myid:] - INFO  [main:PrepRequestProcessor@965] - Shutting down
    [junit] 2016-07-20 13:37:47,970 [myid:] - INFO  [main:SyncRequestProcessor@191] - Shutting down
    [junit] 2016-07-20 13:37:47,970 [myid:] - INFO  [SyncThread:0:SyncRequestProcessor@169] - SyncRequestProcessor exited!
    [junit] 2016-07-20 13:37:47,971 [myid:] - INFO  [main:FinalRequestProcessor@479] - shutdown of request processor complete
    [junit] 2016-07-20 13:37:47,970 [myid:] - INFO  [ProcessThread(sid:0 cport:11222)::PrepRequestProcessor@154] - PrepRequestProcessor exited loop!
    [junit] 2016-07-20 13:37:47,971 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222,name1=InMemoryDataTree]
    [junit] 2016-07-20 13:37:47,971 [myid:] - INFO  [main:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port11222]
    [junit] 2016-07-20 13:37:47,972 [myid:] - INFO  [main:FourLetterWordMain@85] - connecting to 127.0.0.1 11222
    [junit] 2016-07-20 13:37:47,972 [myid:] - INFO  [main:JMXEnv@146] - ensureOnly:[]
    [junit] 2016-07-20 13:37:47,976 [myid:] - INFO  [main:ClientBase@568] - fdcount after test is: 52 at start it was 37
    [junit] 2016-07-20 13:37:47,976 [myid:] - INFO  [main:ClientBase@570] - sleeping for 20 secs
    [junit] 2016-07-20 13:37:47,977 [myid:] - INFO  [main:ZKTestCase$1@65] - SUCCEEDED testQuota
    [junit] 2016-07-20 13:37:47,977 [myid:] - INFO  [main:ZKTestCase$1@60] - FINISHED testQuota
    [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.685 sec

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/ZooKeeper-trunk-jdk8/trunk/build.xml:1233: Tests failed!

Total time: 73 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



###################################################################################
############################## FAILED TESTS (if any) ##############################
1 tests failed.
FAILED:  org.apache.zookeeper.test.WatcherTest.testWatchAutoResetWithPending

Error Message:
For 68 expected:<1> but was:<0>

Stack Trace:
junit.framework.AssertionFailedError: For 68 expected:<1> but was:<0>
	at org.apache.zookeeper.test.WatcherTest.testWatchAutoResetWithPending(WatcherTest.java:215)
	at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:79)